ACR DNS Bouncing: When Geo-Replication Breaks Your CI

We have a container registry that’s replicated across three Azure regions. Push an image, and within a few minutes, all three replicas have it. Sounds great. One problem: “within a few minutes” is not “instantly,” and CI doesn’t wait.

The race condition

Here’s what happens. CI builds an image. Pushes it to maifalcon.azurecr.io. Immediately after, another job pulls the same image to create a multi-arch manifest or copy it to a secondary registry.

maifalcon.azurecr.io is a geo-replicated ACR. That domain name doesn’t point to one server — it’s a global endpoint backed by DNS that can resolve to any replica. You push to westus3. The subsequent pull resolves to eastus2. Eastus2 hasn’t finished syncing yet.

CI job 1: docker push maifalcon.azurecr.io/yolo:abc123
  → DNS resolves to westus3
  → push succeeds

CI job 2 (30 seconds later): docker pull maifalcon.azurecr.io/yolo:abc123
  → DNS resolves to eastus2
  → image not found
  → build fails

The image exists. It’s just not here yet. And “here” changes every time you make a DNS query.

This is a push-then-immediately-pull race condition, and it only affects registries with geo-replication enabled. Our other registries — inf5acr, mangopdx — are single-region and don’t have this problem.

Three workarounds, stacked like geological layers

Over time, the team layered on three independent mitigations. Each one addressed a symptom without fixing the root cause, creating a system that works but looks like it was designed by a committee of people who never talked to each other.

Workaround 1: dnsmasq with a 2-hour TTL

The first fix: pin DNS resolution for the duration of the build. Install dnsmasq on the runner, configure it with min-cache-ttl=7200 (2 hours), and route all DNS through it. Once the runner resolves maifalcon.azurecr.io to a specific IP, every subsequent request during that build hits the same replica.

- name: Setup dnsmasq
  run: |
    sudo apt-get install -y dnsmasq
    echo "min-cache-ttl=7200" | sudo tee /etc/dnsmasq.d/cache.conf
    sudo systemctl restart dnsmasq
    echo "nameserver 127.0.0.1" | sudo tee /etc/resolv.conf

This works. It also means you’re overriding the system DNS resolver on every CI run, which is the kind of thing that causes bizarre debugging sessions six months later when someone adds a step that needs to resolve a different hostname and can’t figure out why DNS is “broken.”

Only applied to the maifalcon pipeline. The other registries don’t need it because they’re not geo-replicated.

Workaround 2: wait-for-image polling

The second fix: before doing multi-arch merges or cross-registry copies, poll the target registry until the image appears. A custom GitHub Action that runs az acr manifest show in a loop with exponential backoff, up to 10 minutes.

- name: Wait for image availability
  uses: ./.github/actions/wait-for-image
  with:
    registry: maifalcon
    image: yolo
    tag: ${{ github.sha }}
    timeout: 600  # 10 minutes

The action keeps hitting the global endpoint. Sometimes it resolves to the replica that has the image on the first try. Sometimes it takes 8 minutes. The timeout is generous because the alternative is a failed build.

This adds 0-10 minutes of latency to every build. Average case is about 2 minutes. Worst case, it times out and the build fails anyway, and someone re-runs it.

Workaround 3: aggressive retries

The third fix: just retry everything. The docker buildx imagetools commands that create multi-arch manifests get 12 attempts with 3-minute delays between each.

- name: Create multi-arch manifest
  uses: nick-fields/retry@v3
  with:
    max_attempts: 12
    retry_wait_seconds: 180
    command: |
      docker buildx imagetools create \
        --tag maifalcon.azurecr.io/yolo:${{ github.sha }} \
        maifalcon.azurecr.io/yolo:${{ github.sha }}-amd64 \
        maifalcon.azurecr.io/yolo:${{ github.sha }}-arm64

Twelve attempts times three minutes is 36 minutes of maximum retry budget. For a single step. In a build that’s already 40 minutes long.

The math here reveals the desperation. Nobody sets max_attempts: 12 because they think it’ll take 12 tries. They set it because they’ve seen it take 8 and didn’t want to find out what happens at 9.

The cumulative cost

Stack these three workarounds together:

On a good day, everything syncs quickly and the workarounds are invisible. On a bad day — maybe a region is under load, or replication is slow — the build takes 40 minutes of actual work plus 15 minutes of waiting and retrying. And the CI logs are a wall of “attempt 4 of 12, waiting 180 seconds.”

Three independent mitigations for one root cause. The classic.

The actual fix: regional endpoints

ACR supports regional endpoints. Instead of hitting the global DNS endpoint that might resolve to any replica, you talk directly to a specific region:

Global:   maifalcon.azurecr.io
Regional: maifalcon.westus3.geo.azurecr.io

Push to maifalcon.westus3.geo.azurecr.io. Pull from maifalcon.westus3.geo.azurecr.io. Same region, same replica, no race condition. The image is there because you just put it there.

The regional endpoint format is <registry>.<region>.geo.azurecr.io. For maifalcon, the primary region is westus3. For maimanifold, it’s westus2. These are the regions where the registry was created — the “home” replica.

CI only, not runtime

Here’s an important architectural decision: regional endpoints only get used in CI. Runtime constants — the image URLs that researchers’ training jobs use to pull images — stay on the global endpoint.

Why? The push-then-immediately-pull race condition only exists in CI, where one job pushes and the next job pulls within seconds. Runtime pulls happen minutes to hours after the push, by which time replication has completed. The race window is closed.

And changing runtime would mean updating 61+ files across the codebase — every place that constructs an image URL from the registry constants. The risk-reward ratio is terrible. Fix the 3 CI files that have the actual problem, leave the 61 runtime files alone.

The implementation plan

The migration from three workarounds to one clean fix has nine steps:

1. Add regional endpoint env vars — MAIFALCON_REGIONAL_URL=maifalcon.westus3.geo.azurecr.io
2. Update ACR login — az acr login with the regional endpoint
3. Use regional tags in Docker Bake — push targets use the regional URL
4. Update multi-arch merge — imagetools create uses regional URL for both source and destination
5. Remove dnsmasq steps — no longer needed when DNS doesn’t bounce
6. Remove wait-for-image action — no replication delay when staying in one region
7. Reduce retries — from 12 attempts/180s delay to 3 attempts/30s delay (keep minimal retry for transient network errors)
8. Keep runtime constants unchanged — constants.py still uses the global endpoint
9. Don’t delete the action code yet — keep wait-for-image and dnsmasq configs around as fallback until the regional approach proves stable in production

Steps 5-7 are the satisfying ones. Deleting workaround code is always more fun than writing it.

The pattern

This is a recurring pattern in infrastructure work. A root cause produces intermittent failures. Each failure gets a targeted workaround. The workarounds accumulate. Eventually someone looks at the stack of band-aids and realizes the root cause has a clean fix that makes all of them unnecessary.

The three workarounds weren’t wrong when they were added. Dnsmasq was a quick win. Wait-for-image was defensive. Retries were pragmatic. Each one bought time. But they were treating DNS resolution as an unpredictable force of nature instead of asking: “can we just… not have this DNS bounce in the first place?”

Regional endpoints were available the whole time. The documentation just doesn’t scream “use this to fix your CI race condition” — it talks about latency optimization and disaster recovery. The connection between “geo-replicated ACR” and “CI push-then-pull race” isn’t obvious until you’ve lived through the debugging.

Sometimes the fix isn’t better retry logic. It’s removing the reason you needed retries.