We had a CI workflow that pushed container images to Azure Container Registry, then immediately tried to pull them back. Sounds fine. Except it failed — randomly, intermittently, infuriatingly. The image was there. And also not there. Depending on who you asked.
The problem was DNS bouncing on a geo-replicated registry. Fixing it took one feature flag and five follow-up commits. Along the way I got a nice tour of how DNS, container registries, and distributed replication actually interact.
One name, many backends
Here’s the setup. Azure Container Registry supports geo-replication — you have one registry name (maifalcon.azurecr.io) but the data is replicated across multiple regions (say, West US 2, West US 3, East US). When you resolve that hostname, DNS returns an IP for one of those replicas. Which one? Depends on your location, DNS caching, load balancing policy, and the phase of the moon.
This is a standard pattern in distributed systems. It’s how CDNs work, how global databases route traffic, how cloud services give you “one endpoint” that actually fans out to a fleet. The abstraction is beautiful — you don’t care where the data lives, you just talk to maifalcon.azurecr.io and the system figures it out.
Until the abstraction breaks.
The read-your-own-writes problem
Geo-replication is eventually consistent. You push an image to replica A, and it takes some time — seconds, sometimes minutes — to propagate to replicas B and C. During that window, the image exists on A but not on B.
Now here’s where DNS makes it worse. Your CI runner pushes an image. The push goes to whatever IP DNS resolved to — let’s say replica A in West US 3. Great, push succeeds. Next step: create a multi-arch manifest that references that image. The runner does another DNS lookup for maifalcon.azurecr.io. This time, DNS returns replica B in East US. Replica B hasn’t received the image yet.
404. Image not found.
This is a classic read-your-own-writes violation. In database terms, you wrote a row and immediately read it back, but your read went to a different replica that hasn’t caught up. The system is eventually consistent, but “eventually” isn’t good enough when your next CI step runs 2 seconds later.
The subtle thing: the client didn’t do anything wrong. It talked to the same hostname both times. The inconsistency is invisible at the application layer — it’s hiding in the DNS resolution, one layer below where you’re looking.
DNS: a caching layer you forgot about
Quick DNS refresher, because it matters here.
When your machine resolves maifalcon.azurecr.io, the answer comes with a TTL (time to live). Your local resolver caches that answer for TTL seconds. After that, next lookup goes back to the authoritative nameserver, which might return a different IP.
For a geo-replicated service, the authoritative DNS is doing some form of traffic management — GeoDNS, weighted round-robin, latency-based routing, whatever. The point is: two lookups separated by more than the TTL can resolve to different backends.
TTLs for cloud services are often short. 60 seconds, sometimes less. That’s by design — they want to shift traffic quickly during failovers or load rebalancing. But short TTLs mean your CI runner, making requests minutes apart, is very likely to hit different replicas between steps.
The dnsmasq hack
The previous workaround in our CI was… creative. Install dnsmasq on the runner, configure it as the local DNS resolver, and crank the TTL to 2 hours:
max-cache-ttl=7200
max-ttl=7200
min-cache-ttl=7200
This pins DNS resolution for the entire workflow run. First lookup returns replica A? You’re talking to replica A for the next 2 hours. No bouncing.
It works. But it’s a hack, and a fragile one:
- You’re overriding the system DNS resolver on a CI runner. Things can go wrong.
- If replica A goes down mid-workflow, you’re stuck talking to a dead backend for 2 hours.
- The
min-cache-ttloverride technically violates DNS protocol — the authoritative server set a TTL for a reason, and you’re ignoring it. - You also needed “wait-for-image” polling steps — retry loops that waited up to 36 minutes for replication to finish.
The core issue: you’re fighting eventual consistency at the wrong layer. You want strong consistency for a specific access pattern, but you’re trying to achieve it by hacking DNS instead of asking the data layer for it.
Regional endpoints: the right fix
The real fix is simple in concept: stop using the geo-balanced endpoint. Talk directly to the replica you want.
ACR’s regional endpoints give you that: maifalcon.westus3.geo.azurecr.io always routes to the West US 3 replica. No DNS load balancing, no bouncing. Push to West US 3, pull from West US 3 — you’re talking to the same replica both times.
This is a pattern you see everywhere in distributed systems. The global endpoint gives you convenience and automatic failover. The regional endpoint gives you consistency and predictability. Most of the time you want the former. But when you need read-your-own-writes — in CI, in deployment pipelines, in anything that does write-then-read in quick succession — you want the latter.
It’s the same trade-off as reading from a database primary vs. a read replica. The replica gives you scale. The primary gives you recency. Pick the one that matches your access pattern.
The five-commit spiral
The actual fix was one line — swap maifalcon.azurecr.io for maifalcon.westus3.geo.azurecr.io and use az acr login --endpoint westus3. Ship it.
Except regional endpoints are a preview feature. And preview features come with surprises. What followed was a five-commit spiral that’s honestly a pretty good case study in how CI work actually goes:
Commit 1: The feature. Use regional endpoints, remove dnsmasq, remove wait-for-image polling. Clean diff, -125 lines. Beautiful.
Commit 2: az acr login --endpoint requires a CLI extension that isn’t installed on runners. Pivot to --expose-token + docker login as a workaround. Also discovered that one of our registries doesn’t have regional endpoints enabled yet — revert that one to non-regional.
Commit 3: Actually, just install the extension. Download the .whl file, az extension add. Revert the expose-token workaround.
Commit 4: az extension add prompts for confirmation in non-TTY environments. CI runners don’t have a TTY. Add --yes.
Commit 5: az extension add parses the filename to determine the extension name. The generic filename we used didn’t match what it expected. Use the full wheel filename.
Commit 6: Reviewer asks for retries on the download step. Use Wandalen/wretry.action to be consistent with the rest of the workflow.
One feature, five fixes. That’s a 1:5 ratio, and honestly that’s normal for CI work. Each fix is trivial in isolation — a flag here, a filename there. But you can’t predict them from your laptop. They only surface when code meets the actual runner environment.
Lessons for distributed systems
This whole saga is a nice microcosm of distributed systems concepts:
Eventual consistency has a blast radius. It’s fine for most reads. It’s not fine for the reads that happen 2 seconds after a write in an automated pipeline. Know which reads matter.
DNS is a distributed system too. We think of it as “name → IP” translation, but it’s really a globally distributed, eventually consistent, hierarchically cached key-value store. It has all the same consistency challenges as any other distributed system. When your application’s correctness depends on DNS resolving to the same backend across requests, you have a consistency requirement that DNS wasn’t designed to provide.
Abstractions leak in CI. The geo-replicated ACR abstraction works great for humans — you push from one region, pull from another, and replication happens in the background fast enough that nobody notices. CI is not “nobody.” CI notices. CI does write-then-read in 2 seconds and expects consistency. When your abstraction assumes human-speed access patterns, machines will find the leaks.
Pin what matters. The fix pattern is general: when you need consistency, stop going through the load balancer. Talk to a specific replica. In databases, read from the primary. In registries, use a regional endpoint. In any distributed system, there’s usually a way to bypass the eventual-consistency abstraction when you need to — you just pay for it in reduced availability or flexibility.
Preview features cost more than you think. Not in money — in integration friction. The extension wasn’t installed. The install was interactive. The filename format was wrong. Each is a 30-second fix, but each requires a CI round-trip to discover. Budget for the iteration.