Docker Builds: How Yolo Tags Images and the Stale Prepuller

Yolo builds a new Docker image roughly every two hours. Each image gets two tags: one that’s human-readable, one that’s content-addressed. Then the workflow auto-bumps constants files in yolo itself, commits back to the PR branch, and calls it a day.

That self-bump mechanism? It’s the old system. It’s being replaced. And the thing replacing it is better in almost every way – except it still doesn’t solve the problem I actually ran into, which is a Kubernetes DaemonSet in a completely different repo pre-pulling an image from November 2025. Three months stale. Nobody noticed.

This is a story about two tagging systems, a legacy consumption chain, the new system designed to replace it, and the gap that exists no matter how clever your image pipeline is – the gap at the repo boundary.

The dual-tag system

Every yolo image build produces two tags per image:

Tag Type	Format	Example	Purpose
Git SHA	7-char truncation of `github.sha`	`c1b4b63`	Human-readable, maps to a commit
Content hash	`yolo-${SHA256}`	`yolo-a1b2c3d4e5...`	Deduplication – only changes when the image actually changes

The git SHA tag is straightforward. The workflow’s compute_vars job takes the commit SHA, chops it to 7 characters, and passes it downstream:

GIT_SHA="${{ github.sha }}"
echo "GIT_SHA=${GIT_SHA:0:7}" >> "$GITHUB_OUTPUT"

The content hash is more interesting. Yolo has a custom Python tool called paths_hash that computes a SHA-256 over a specific set of source files:

FILES = {
    "yolo": [
        "mai_kernels/**/*.cu",
        "mai_kernels/**/*.cpp",
        "mai_kernels/**/*.h",
        "mai_kernels/**/*.hpp",
        "mai_kernels/CMakeLists.txt",
        "docker/pretraining/Dockerfile",
        "docker-bake.hcl",
        ".github/workflows/build_yolo_base_and_rocket_containers.yml",
    ],
}

It also resolves the full dependency tree for the mai_job package via uv export, hashes the resulting requirements, and folds that into the same digest. So the hash captures: CUDA/C++ source files, the Dockerfile, the Bake definition, the workflow file itself, and the entire resolved dependency graph.

paths_hash is entirely Sriram’s work for the content-hash tagging system. It lives at lib/paths_hash/ in the yolo monorepo, owned by the DevEx squad (sq-sc-devex in OWNERS). The CLI entrypoint: uv run python -m paths_hash.cmdline yolo computes the hash for the yolo image; paths_hash.cmdline rocket does the same for rocket images. If any of the hashed inputs change — a CUDA kernel, the Dockerfile, a pip dependency — the hash changes. If nothing changes, same hash, same tag, build skipped via the probe-hash mechanism.

Both tags get applied in docker-bake.hcl:

target "pretraining" {
  tags = compact([
    "${REGISTRY_NAME}.azurecr.io/yolo/pretraining:latest",
    "${REGISTRY_NAME}.azurecr.io/yolo/pretraining:${GIT_SHA}",
    "${REGISTRY_NAME}.azurecr.io/yolo/pretraining:yolo-${YOLO_HASH}",
  ])
}

The compact() call drops empty strings – if no hash is provided, the content-hash tag is omitted. Same pattern for rocket images, just with a rocket-${ROCKET_HASH} prefix.

(.hcl = HashiCorp Configuration Language. Docker Buildx adopted it for docker-bake.hcl — a declarative build file that defines multiple image targets, their Dockerfiles, tags, build args, and dependencies between targets. Think of it like a Makefile for Docker builds: instead of running 5 docker build commands with different flags, you define all targets in one file and run docker buildx bake pretraining. The compact() function is HCL’s way of filtering out empty strings from a list. The inherits and contexts fields let targets build on each other — pretraining inherits cache config from docker-build-cache-config-action and uses the output of the shared target as a build context for multi-stage builds.)

Content-addressed builds: the probe-hash trick

Here’s where the content hash pays off. Before building anything, the workflow runs a probe-hash job:

- name: Probe if yolo image with source hash exists
  run: |
    IMAGE_NAME="inf5acr.azurecr.io/yolo/pretraining:yolo-${{ needs.compute_hash.outputs.yolo_hash }}"
    if docker manifest inspect ${IMAGE_NAME}; then
      echo "Image with source hash exists"
      echo "yolo_image_exists=true" >> "$GITHUB_OUTPUT"
    fi

If the tag yolo-${HASH} already exists in the registry, the image content hasn’t changed since last build. Skip the entire 30-60 minute build.

This is content-addressed image building. You commit a README change, the workflow fires on schedule, computes the hash, finds the existing tag, and does nothing. You change a CUDA kernel file, the hash changes, no matching tag exists, and a full rebuild kicks off.

The git SHA tag changes on every single commit. The content hash only changes when something that actually affects the image changes. This distinction matters a lot more than you’d think.

The old way: self-bump to constants.py

After all the images are built, pushed to three ACR registries, and converted to enroot squashfs files for the HPC clusters, the workflow runs one more job: commit_and_update_tags.

This job checks out the PR branch, runs a bunch of sed commands, and commits:

- name: Update Base image references
  run: |
    sed -i "s|^DEFAULT_YOLO_BASE_IMAGE = \".*\"|DEFAULT_YOLO_BASE_IMAGE = \"/mnt/vast/images/yolo/${ENROOT_IMAGE}\"|" \
      mai_cluster/src/mai_cluster/constants.py

    sed -i "s|^DEFAULT_YOLO_BASE_IMAGE_FALCON = \".*\"|DEFAULT_YOLO_BASE_IMAGE_FALCON = \"${FALCON_IMAGE}\"|" \
      mai_cluster/src/mai_cluster/constants.py

    sed -i "s|^DEFAULT_YOLO_BASE_IMAGE_MANIFOLD = \".*\"|DEFAULT_YOLO_BASE_IMAGE_MANIFOLD = \"${MANIFOLD_IMAGE}\"|" \
      mai_cluster/src/mai_cluster/constants.py

The constants files end up looking like this:

DEFAULT_YOLO_BASE_IMAGE = "/mnt/vast/images/yolo/mai+yolo+ngc-25.03-c1b4b63.sqsh"
DEFAULT_YOLO_BASE_IMAGE_FALCON = "maifalcon.azurecr.io/yolo/pretraining:c1b4b63"
DEFAULT_YOLO_BASE_IMAGE_MANGO = "mangopdx.azurecr.io/yolo/pretraining:c1b4b63"
DEFAULT_YOLO_BASE_IMAGE_MANIFOLD = "maimanifold.azurecr.io/yolo/pretraining:c1b4b63"

After the commit, it slaps an image-exported label on the PR. Subsequent pushes to the same branch check for this label and skip the build if the source hash matches.

This is a nice closed loop. Within the yolo repo, image references stay in sync with the actual built images. Nobody needs to remember to update the constants. The workflow builds the image and writes the receipt into the code in the same breath.

But it has problems.

What’s wrong with the self-bump

The sed + commit-back mechanism works, but it creates friction that scales poorly. Two big ones:

Redundant CI runs. The bot commits to the PR branch. That commit triggers CI again. CI sees a new push, runs the full test suite, and the image build workflow fires again too – where it (hopefully) hits the probe-hash cache and skips the actual build. But you still pay for the CI cycle: checkout, setup, hash computation, probe, all the overhead jobs. Every image build creates a ghost CI run that does nothing useful.

The two-PR merge conflict. This is the subtle one. Say PR1 changes uv.lock (adds a new dependency). The image build runs on PR1’s branch, builds an image with PR1’s lock file, and sed-bumps the constants to point at that image. PR2, independently, also changes uv.lock. Same thing happens – image builds, constants bumped.

Now PR1 merges to main. Fine. Main has PR1’s uv.lock, and the constants point to an image built from PR1’s lock file. Everything matches.

Then PR2 merges. Main now has both PR1 and PR2’s uv.lock changes combined. But the constants point to the image built from PR2’s branch – which only had PR2’s changes. No image exists for the PR1+PR2 combination. The constants in main now reference an image that was built from a different dependency set than what main actually has.

This is a time bomb. It might not explode immediately – the dependency difference might be minor, or the next scheduled build on main will produce a correct image and bump the constants again. But there’s a window where the constants lie. And if something goes wrong in that window, good luck debugging it.

The new way: content-hash image tags

Ryan designed a new system to replace the self-bump. The idea: instead of having CI write image references into source code, have the CLIs compute the content hash themselves and look up the image directly.

The design is called “Image builds and tags.” The key insight: the content hash is deterministic. Given the same source files and dependency graph, any machine can compute the same hash. So instead of CI telling you “here’s your image tag” via a committed constant, the CLI just… computes the hash and asks the registry “do you have this?”

This eliminates the bot commits entirely. No more sed to constants.py. No more redundant CI runs triggered by the bot push. No more merge conflict window where the constants lie.

And it solves the two-PR problem cleanly. When PR1+PR2 both merge to main, the next workflow run on main computes the hash from the combined state (both uv.lock changes), builds an image for that exact combination if one doesn’t exist, and tags it. Meanwhile, any CLI running on main computes the same hash locally and finds the right image. No constants file as middleman. No window where the reference is stale.

Sriram is rolling this out in phases:

Phase	What	Status
a) Compute hashes + tag images	Build workflow computes content hash, tags images with it	Done (PR #20682, merged Feb 27)
b) Use hashes in CLIs	CLIs compute hash locally, look up image by content tag	In flight (PR #22286 – `--use-chit` flag for `yolo interactive`)
c) Eliminate extra builds	Skip builds when content-hash tag already exists (the probe-hash logic, but applied to the self-bump decision)	Not started
d) Image sharing use cases	Support scenarios where teams share images across contexts	Not started

Phase (a) is already live – images are getting content-hash tags today. Phase (b) is the big behavioral change: once CLIs use --use-chit by default, the self-bump to constants.py becomes dead code.

Who built paths_hash: Sriram Musiri. PR #20357 (merged Feb 20, 2026) created the package, PR #20682 (merged Feb 27) wired it into the build workflow.

Current state: Both tag systems are active simultaneously. The pretraining target in docker-bake.hcl tags each image with both ${GIT_SHA} and yolo-${YOLO_HASH}. The content hash tag is additive — it doesn’t replace the git SHA tag. constants.py still uses git SHA tags: DEFAULT_YOLO_BASE_IMAGE_FALCON = "maifalcon.azurecr.io/yolo/pretraining:c1b4b63" — those are 7-char git SHAs, updated by the self-bump bot. The prepuller automation uses content-hash tags — yolo-${HASH} format.

The consumers don’t overlap: CLI users go through constants.py (git SHA), the prepuller goes through images.yaml (content hash). Same image, different pointers. Two naming systems pointing at the same artifact, for two consumers with completely different update mechanisms.

The gap that remains

So the new system fixes how users and CLIs find images. Content hash computed locally, image looked up in the registry, no hardcoded constants. Clean.

But there’s a different consumer that neither the old system nor the new system addresses: infrastructure.

Enter the image prepuller.

The prepuller problem

The yolo pretraining image is about 40GB. When a researcher submits a training job to a Kubernetes cluster, the container runtime needs to pull that image before the pod can start. On a cold node – one that’s never pulled this image before – that pull takes minutes. Possibly many minutes. On GPU nodes where you’re paying per-second for $50+/hour hardware, “minutes of idle time pulling a container” is the kind of waste that makes finance people twitch.

The fix is a Kubernetes DaemonSet called the image-prepuller. A DaemonSet is Kubernetes’ way of saying “this thing must run on every matching node” — exactly one pod per node, guaranteed, as opposed to a Deployment which runs N replicas wherever the scheduler feels like putting them. It’s the right primitive for node-level agents: log collectors, monitoring sidecars, and yes, image pre-pullers.

This particular DaemonSet uses nodeAffinity to only target GPU nodes (Standard_ND128isr_NDR_GB200_v6). No point pre-pulling 40GB ML images on CPU nodes that will never run training jobs. The main container is literally pause — a 740KB image that does nothing. It exists solely to keep the pod alive so Kubernetes considers the DaemonSet “running.”

The real work happens in initContainers — containers that run to completion before the main container starts. Each initContainer references one image, echoes “done”, and exits. The pull is the side effect. kubelet downloads the image to the node’s local image store, and it stays there for any future pod that needs it. Cheap trick, but it works.

The lifecycle is automatic: when a new node joins the cluster and matches the affinity selector, the DaemonSet controller creates a pod on it, which triggers the initContainers, which pulls the images. When the DaemonSet spec changes (new image tag in images.yaml), Kubernetes performs a rolling update — deleting old pods and creating new ones, which re-triggers the initContainers to pull the updated images. So the chain is: images.yaml updated → ArgoCD syncs → K8s rolling update → new images pulled on every GPU node.

The prepuller lives in the mango repo – the infrastructure repo, separate from yolo. It’s deployed via ArgoCD, which watches a git branch and auto-syncs changes to the clusters. The consumption chain looks like this:

Yolo CI builds image
  --> pushed to maifalcon.azurecr.io/yolo/pretraining:${TAG}
                        |
                        v (manual PR -- this is the gap)
                        |
mango repo: applications/image-prepuller/overlays/falcon-base/images.yaml
                        |
                        v (kustomize strategic merge patch)
                        |
mango repo: applications/image-prepuller/base/daemonset.yaml
                        |
                        v (ArgoCD auto-sync from git HEAD)
                        |
Falcon K8s clusters: DaemonSet on GPU nodes
                        |
                        v (initContainers pull images)
                        |
Node local cache: images ready for workloads

How kustomize builds the DaemonSet

The prepuller’s directory structure in mango looks like this:

applications/image-prepuller/
├── application-falcon-phx-ga.yaml     # ArgoCD Application — tells ArgoCD WHERE to find the kustomize overlay
│                                       # points to: applications/image-prepuller/overlays/falcon-base
│                                       # syncs to: kube-system namespace
├── base/
│   ├── kustomization.yaml             # Lists resources: just daemonset.yaml
│   └── daemonset.yaml                 # The "template" DaemonSet — nodeAffinity, pause container, placeholder initContainers
├── overlays/
│   └── falcon-base/
│       ├── kustomization.yaml         # Inherits ../../base, applies patches: images.yaml
│       └── images.yaml                # JSON patch that REPLACES initContainers with actual images to pull

Kustomize is Kubernetes’ built-in config management tool. Instead of templating (that’s Helm’s thing — variables, conditionals, Go templates), kustomize overlays patches on top of a base manifest. No variables, no templates — just “start with this YAML, apply these patches.”

The base is the DaemonSet skeleton: node affinity rules, GPU node tolerations, the pause main container. Doesn’t change often. The overlay is per-cluster customization — falcon-base is the Falcon cluster’s overlay. Different clusters can have different overlays pointing to different registries or different image lists.

The patch (images.yaml) is a JSON Patch (op: replace) that swaps out the entire initContainers list. “Strategic merge patch” is the Kubernetes term for patches that merge intelligently by matching on keys — but this one uses op: replace, which is simpler: wholesale replacement of the initContainers array. Old list gone, new list in.

ArgoCD reads application-falcon-phx-ga.yaml, sees it points to the overlays/falcon-base path, runs kustomize build to merge base + overlay, and applies the result to the cluster. When the git source changes (someone updates images.yaml), ArgoCD detects the diff and auto-syncs. The whole thing is declarative — the desired state lives in git, ArgoCD makes the cluster match.

The images.yaml file is a kustomize JSON patch that overrides the base DaemonSet’s initContainers with the actual images to prepull:

- op: replace
  path: /spec/template/spec/initContainers
  value:
    - name: prepull-yolo-pretraining
      image: maifalcon.azurecr.io/yolo/pretraining:6305057
      command: ["echo", "Image prepulled successfully"]

See that tag? 6305057. A git SHA from November 2025.

The yolo workflow builds a new image every two hours on main. Each one gets pushed to the Falcon ACR with a fresh tag. But the prepuller just sits there, faithfully pre-pulling the November image on every new node, while the actual workloads pull whatever version the researcher requested – usually something much newer.

Three months of images built. Zero updates to the prepuller. The automation stopped at the repo boundary.

And yes — the prepuller was running the entire time. DaemonSets don’t stop because their config is stale. Every new GPU node that joined the cluster dutifully pulled the November image. But researchers don’t use the prepuller’s image directly. They specify their own image tag when launching jobs — via constants.py defaults or CLI flags. So the prepuller was pulling one image, and the actual workloads were pulling a different one.

Net effect: the prepuller was doing work — downloading 30GB+ on every new node — but the wrong 30GB. The image it cached wasn’t the one jobs actually used. Cold starts weren’t eliminated, just… mildly warmed. Some layers overlap between old and new images, so partial cache hits still helped. But you don’t get the full benefit of pre-pulling when you’re pre-pulling something nobody uses anymore.

People noticed because HPC was scaling to 512+ GPU nodes for large training runs. At that scale, cold-start latency across hundreds of nodes becomes painfully visible. Soumyadeep explicitly asked if the prepuller was even available on Nebius nodes — it wasn’t yet. Nathan deployed it. That conversation is what kicked off the “wait, is the prepuller even pulling the right image?” thread.

Sriram’s content-hash work doesn’t help here. The prepuller doesn’t run a CLI. It doesn’t compute hashes. It’s a DaemonSet with a hardcoded image reference in a YAML file in a different repo. It needs someone – or something – to update that YAML file when a new image is available.

This is a fundamentally different consumption pattern. CLIs are active consumers: they compute, query, decide. Infrastructure is a passive consumer: it gets configured, and it stays configured until someone changes it.

The stable tag debate

Before jumping to the fix, a natural question came up: “Can’t we just use a stable tag like latest for the prepuller?”

Ajinkya asked this. Yiran said no. And Yiran was right.

Stable tags are mutable. latest today is a different image than latest yesterday. If a node pulls latest at 2pm and another pulls it at 4pm, they might get different images. If something breaks, you can’t tell which latest they had. You’re debugging without a fingerprint.

Content hashes are the answer: they’re immutable and meaningful. A content hash tag points to exactly one image, forever. If a node has yolo-a1b2c3d4, you know precisely what’s on it. You can reproduce the environment, diff it against another node, trace it back to the source files that produced it.

For the prepuller, this means: use content-hash tags, update them via automation, and you get immutability (debuggable) plus freshness (automated). No mutable latest tag that you can’t reason about.

Three consumption paths

So we end up with three distinct ways images get consumed, each solving a different problem:

Legacy (being phased out):

Yolo CI --> sed updates constants.py --> user reads constant --> launches job

Simple but fragile. Bot commits, merge conflicts, stale-by-design between builds.

New – Sriram’s content-hash CLIs (shipping now):

Yolo CI --> tags image with content hash
CLI     --> computes hash locally --> looks up image in registry --> launches job

No bot commits, no constants file as middleman, correct by construction. Solves the user-facing consumption problem.

Prepuller – cross-repo automation (my work):

Yolo CI --> repository_dispatch to mango --> updates images.yaml --> ArgoCD syncs

Solves the infrastructure consumption problem. Carries the content-hash tag across the repo boundary so the prepuller always has a fresh, meaningful, immutable reference.

Each path exists because the consumer is different. CLIs can compute. Infrastructure can’t. And the old system tried to make one mechanism (committed constants) serve both, which worked for neither particularly well.

The fix: closing the loop

For the prepuller specifically, repository dispatch is the right answer. After building, yolo fires a repository_dispatch event to the mango repo. Mango has a workflow that catches it, updates images.yaml, commits, and pushes. ArgoCD picks up the change.

# In yolo's workflow, after successful build:
- name: Notify mango of new image
  uses: peter-evans/repository-dispatch@v3
  with:
    token: ${{ secrets.CROSS_REPO_TOKEN }}
    repository: infinity-microsoft/mango
    event-type: yolo-image-updated
    client-payload: |
      {
        "tag": "${{ needs.compute_vars.outputs.git_sha }}",
        "content_hash": "yolo-${{ needs.compute_hash.outputs.yolo_hash }}",
        "registry": "maifalcon.azurecr.io"
      }

On the mango side, a receiving workflow catches yolo-image-updated, writes the new content-hash tag into images.yaml, commits, and pushes. ArgoCD sees the commit, syncs the DaemonSet, and the prepuller starts pulling the new image on every GPU node.

The content-hash tag is the right one to use here – same reasoning as the CLIs. The prepuller only restarts its pods when the image actually changes. No restart for a README commit. No pod churn for a CI-only file change.

The monitoring gap

One thing this whole exercise revealed: we have zero visibility into image pull performance. We don’t know:

How long it takes to pull the 40GB image on a fresh node
Whether the prepuller is actually keeping up with new images
How image size trends over time (is it growing? shrinking?)
Whether nodes are hitting ACR rate limits during pulls

The prepuller runs, the pods report “Image prepulled successfully,” and we trust that echo statement as our only signal. There’s no Datadog metric for pull duration, no alert for a prepuller pod stuck in ImagePullBackOff, no dashboard showing which nodes have which image versions cached.

This is a different problem from the staleness issue, but they’re related. If we had visibility into pull duration, someone would have noticed that node cold-start times weren’t improving – because the prepuller was pulling an image nobody was using anymore.

The meta-lesson

Yolo’s image pipeline is, by most standards, impressively automated. Content-addressed builds. Multi-registry push. Multi-arch manifests. Automatic constant bumping (soon to be retired). Dedup. Retry logic. Cache management. The whole thing is about 1,700 lines of workflow YAML across multiple files, and it works reliably.

And it’s getting better. The content-hash CLI work eliminates an entire class of problems – bot commits, merge conflicts, stale constants. It’s a genuinely better architecture.

But both the old system and the new system share the same blind spot: they optimize for consumption within the repo boundary. The self-bump updates constants inside yolo. The content-hash CLI computes hashes inside the user’s checkout. Neither reaches across to the infrastructure repo where the prepuller lives.

This is true of most build systems I’ve seen. The build is automated. The deployment is automated. The bridge between them – the part where Repo A’s output becomes Repo B’s input – is a human with a clipboard.

The content hash is the building block for fixing this everywhere. It’s a stable, meaningful identifier that changes exactly when you’d want a downstream consumer to notice. Whether the consumer is a CLI computing the hash locally, or a cross-repo workflow carrying it to a YAML file in another repo – the identifier is the same. The mechanism for delivering it is what varies.

Three months is a long time for a prepuller to fall behind. But at least now there’s a plan where it can’t happen again – and the plan doesn’t involve sed committing to constants.py.