I Thought It Was Just One Tag: Docker Image Tagging Rabbit Hole

2026/03/12

BUILDazurecidockerkubernetes

I wanted to automate a simple thing: when the image build finishes, update the prepuller with the new tag. Two repos, one webhook, done by lunch.

I was not done by lunch.

The prepuller, briefly

A prepuller is a Kubernetes DaemonSet whose only job is to pull container images onto every node before anyone needs them. It runs initContainers that docker pull the images and then exit. When a real workload starts on that node, the image is already cached locally. Instead of waiting 5 minutes for a 40GB image to download, the pod starts in seconds.

Ours lives in the mango repo (infra/GitOps). It has an images.yaml that lists the image references — registry, repository, tag. ArgoCD watches the repo. Change the tag in images.yaml, ArgoCD syncs, the DaemonSet rolls, new images get pulled to every node. Clean.

The automation I wanted:

  1. Yolo repo (monorepo): image build finishes → fire a repository_dispatch event to mango with the new image tag
  2. Mango repo: receive the dispatch → update images.yaml → commit to main → ArgoCD syncs

Two steps. One tag. What could be complicated about that?

Choosing the tag

My first design decision: which tag to send? The build produces two kinds:

Tag format Example Changes when…
Git SHA d492f8f Every commit, even if the image content is identical
Content hash yolo-03000f1e7a11... Only when the source files that feed the Docker build actually change

Content hash is clearly better for a prepuller. The whole point of prepulling is caching — you want the DaemonSet to restart only when the image actually changed. If you use the git SHA, every merge to main rolls the prepuller, even if someone just edited a README. The pods restart, the initContainers re-pull an image that’s byte-for-byte identical to what’s already cached, and you’ve churned every node in the cluster for nothing.

Content hash. Obviously. I wrote the dispatch event to send yolo-{content_hash} and moved on.

The verification that wasn’t

Before wiring up the mango side, I wanted to verify the tag actually exists in the target registry. Basic due diligence. The prepuller pulls from maifalcon.azurecr.io, so let me just check that the content-hash tag is there:

az acr manifest show \
  --registry maifalcon \
  --name yolo/pretraining \
  --tag "yolo-03000f1e7a1119c3b329c1a668e280311594b412abc30f75054cd89e71617de6"

Not found.

Huh. The build succeeded. The logs say it pushed the image. The tag exists in the build registry (inf5acr). But maifalcon doesn’t have it.

Let me try the architecture-suffixed version:

az acr manifest show \
  --registry maifalcon \
  --name yolo/pretraining \
  --tag "yolo-03000f1e7a1119c3b329c1a668e280311594b412abc30f75054cd89e71617de6-amd64"

Found.

Wait. There’s a suffixed version? And the bare tag is missing? What is going on?

The tag matrix

I went and read the build workflow. Actually read it, not skimmed-it-and-assumed. Here’s what a single successful build produces:

Per architecture (amd64, arm64):

Multi-arch manifests (points to both architectures):

Across registries:

That’s 4 tag variants × 4 registries = 16 image references from one build. And I was thinking in terms of “a tag.”

But here’s the kicker: the multi-arch manifest — the bare yolo-{content_hash} tag without an architecture suffix — is created by a separate merge step. And that step has a conditional:

- name: Merge and push multi-arch image
  if: steps.check_changes.outputs.should_build == 'true'

If the build determines the image content hasn’t changed since last time, it skips the multi-arch merge. The per-architecture tags get pushed (they’re part of the build step itself), but the bare multi-arch manifest doesn’t get created. The step that creates the tag I wanted to use… might not run.

Everyone else uses git SHA

While debugging this, I noticed something else. The build pipeline has a “Commit & Update Tags” step that writes the new tag back to constants.py — the file that runtime code reads to know which image to pull. That step uses the git SHA, not the content hash.

# constants.py (auto-updated by CI)
YOLO_IMAGE_TAG = "d492f8f"

The job launcher reads this. The resource dispatcher reads this. Every piece of code that constructs an image URL reads constants.py, and constants.py uses the seven-character git SHA.

The content hash? It exists in the build logs. It gets pushed as a tag. But nothing in the runtime system actually references it.

My prepuller automation would be the only consumer of the content-hash tag in the entire system. The odd one out, using a tag format that nothing else uses, referencing a manifest that might not exist because the step that creates it is conditional.

The mental model gap

My mental model going in:

Build → Tag → Push → Done
         ↓
    (one tag, one registry, one architecture)

The actual system:

Build (per arch)
  ├── Push per-arch tags to build registry
  ├── Copy per-arch tags to 3 other registries
  ├── Create multi-arch manifest (conditional!)
  ├── Push multi-arch manifest to all registries (conditional!)
  └── Update constants.py with git SHA

I was thinking about one tag. The system produces a matrix. And the matrix has holes — conditional steps that may or may not fill in certain cells depending on what changed.

This is the gap between “I want to reference an image” and “I understand how images get distributed.” The first is a single string. The second is a directed graph of build steps, copy operations, conditional merges, and registry fanout, where any node in the graph might be skipped.

Why does it get this complicated?

It’s not that anyone designed it to be complicated. Each piece of complexity exists for a good reason:

Multi-arch because the fleet has both amd64 and arm64 nodes. You can’t ship one binary anymore. Every image is really two images wearing a trenchcoat (the multi-arch manifest).

Multi-registry because clusters live in different Azure regions and different trust boundaries. Falcon can’t pull from Mango’s ACR, and you don’t want cross-region pulls adding minutes to pod startup. So the same image lives in four registries.

Content hash because cache stability. The git SHA changes on every commit. The content hash only changes when the Docker build inputs change. For caching purposes — whether it’s a prepuller, a layer cache, or a node’s image cache — content hash is the right identifier.

Git SHA because human readability. d492f8f maps to a commit. You can git log it. You can find the PR. Content hashes are opaque — yolo-03000f1e7a1119c3b329c1a668e280311594b412abc30f75054cd89e71617de6 tells you nothing about when or why that image was built.

Conditional merge because building a multi-arch manifest takes time and involves cross-registry operations that can fail (see the ACR DNS bouncing post). If nothing changed, don’t risk it.

Every decision is locally rational. Stack them together and you get a system where a single build produces 16 image references across a tag×registry×architecture matrix, with conditional steps that may or may not populate certain cells. From the outside it looks like overengineering. From the inside it looks like “well, we need multi-arch, and we need multi-registry, and we need cache stability, and here we are.”

The resolution

I switched the prepuller to use the git SHA tag.

Yes, the same git SHA I spent two sections arguing against. The one that changes on every commit even when the image is identical. The “obviously worse” choice for cache stability.

Here’s why: the git SHA tag is the one that everything else uses. It’s the tag that constants.py stores. It’s the tag that the multi-arch merge step always creates (it’s not conditional on content changes — it runs whenever the build runs). It’s the tag that exists in every registry, as a multi-arch manifest, every time.

The content hash is technically superior for the prepuller’s purposes. But it’s also:

Sometimes the technically correct choice is the wrong engineering choice. The git SHA tag is reliable, well-understood, consistently available, and the same tag format every other consumer uses. The prepuller will restart a few extra times on no-op commits. That’s fine. Unnecessary restarts are annoying. Missing tags are broken.

The iceberg

Every time I automate something in this build system, I discover the same thing: the thing I’m automating is the tip. Underneath is a surprisingly deep system of moving parts that all need to agree.

“Just update a tag” requires understanding:

None of this is written down as a cohesive document. It’s distributed across workflow YAML files, Dockerfile targets, a Docker Bake file, constants.py, and the tribal knowledge of whoever set up the pipeline two years ago. You learn it by trying to do something simple and discovering why it isn’t.

This is, I think, the defining experience of infrastructure work. You arrive with a model that has three boxes and two arrows. You leave with a model that has sixteen boxes, conditional edges, and a matrix you need to keep in your head. And the thing you built still looks like two steps from the outside.

The prepuller automation works now. Two repos, one webhook, one git SHA tag. Done before dinner, not lunch. And I know more about our image distribution pipeline than I ever planned to, which is how infrastructure knowledge always gets acquired — not by reading docs, but by trying to connect two things and finding out what’s in between.