Docker Builds: Container Image Tagging and the Danger of latest

Everyone knows “don’t use latest in production.” It’s the first thing you learn about Docker tagging. But for ML training images, latest isn’t just bad practice — it’s dangerous in a way that regular web services never have to think about. The reason is CUDA kernel coherence, and it turns a tagging mistake from “oops, wrong version” into “silently wrong training results.”

The coherence constraint

ML training images contain pre-compiled CUDA kernels. These are GPU machine code — compiled against a specific version of the Python code that calls them, a specific CUDA toolkit version, and specific GPU architecture flags.

When you compile a custom CUDA kernel (like Flash Attention or our internal mai_kernels), the compiler bakes in assumptions about:

• Tensor memory layouts — how data is arranged in GPU memory
• Launch configurations — thread block sizes, shared memory allocation
• API contracts — function signatures, struct definitions

If the Python code that calls these kernels changes — say, someone refactors a tensor shape or changes an argument order — but the compiled kernel inside the image doesn’t match, you get one of three outcomes:

1. CUDA error at runtime — the lucky case, you get an explicit crash
2. Silent numerical errors — the unlucky case, training proceeds but produces garbage gradients
3. Intermittent NaN — the worst case, training looks fine for hours then explodes

Option 2 is why this matters more than normal Docker tagging hygiene. A web service with the wrong image version returns a 500 error. An ML training job with mismatched CUDA kernels might train for 48 hours on 256 GPUs before someone notices the loss curve looks wrong. That’s real compute dollars evaporating.

The tagging strategy

Our image pipeline uses four tag types:

The SHA tag is the source of truth. It’s a git commit hash, which means you can always map an image back to the exact code that produced it. git show abc123def tells you what CUDA kernel source was compiled, what Python code was baked in, what uv.lock was used. Perfect reproducibility.

The latest tag is a convenience alias that points to the most recent build. It mutates on every CI run. If two PRs merge within minutes of each other, latest flips between them. If a build fails halfway through, latest might point to a partially-pushed image. It’s a moving target with no guarantees.

The main tag is the interesting one.

The trust signal

main only gets applied after GPU tests pass. The CI pipeline looks like this:

PR merges → build image with SHA tag → push to registries
  → run GPU tests against SHA-tagged image
  → all GPU tests pass?
    → yes: tag the image as 'main'
    → no: 'main' stays on the previous passing image

This means main is a trust signal. It says: “this image has been validated on actual GPUs, with actual training workloads, and the CUDA kernels produce correct results.” It’s not just “latest build” — it’s “latest verified build.”

If a PR introduces a CUDA kernel compilation bug, the SHA-tagged image gets built and tested. GPU tests fail. The main tag doesn’t move. Researchers pulling main continue getting the last known-good image. The failure is contained to CI.

This is why latest is dangerous specifically for ML images. latest moves unconditionally — build succeeds, tag moves. main moves conditionally — build succeeds AND tests pass, then tag moves. The gap between those two policies is the gap between “probably fine” and “verified correct.”

How images reach training jobs

Researchers don’t specify image tags. They shouldn’t have to. The system handles it:

1. CI builds image, pushes with SHA tag
2. CI updates constants.py with the new SHA tag
3. CI commits constants.py back to main
4. Researcher calls the job launcher
5. Job launcher reads constants.py → gets current SHA tag
6. resource_request.py dispatches to the right registry based on cluster type:

def get_image_url(resource_request):
    if resource_request.is_mango:
        return f"mangopdx.azurecr.io/yolo:{CURRENT_SHA}"
    elif resource_request.is_falcon:
        return f"maifalcon.azurecr.io/yolo:{CURRENT_SHA}"
    # ... etc

The dispatcher knows which cluster the job targets and picks the nearest registry. Same image contents, different pull location. The SHA tag ensures coherence — the image on mangopdx with tag abc123 contains exactly the same bits as the one on maifalcon with the same tag.

No human picks a tag. No human picks a registry. The system encodes these decisions in constants.py and resource_request.py, and the right thing happens automatically.

When you’d manually update

Three situations where someone touches constants.py by hand:

Cherry-picks without image update. Someone cherry-picks a non-image-affecting change (docs, CI config) and the auto-commit from CI doesn’t trigger because the image hash conditions weren’t met. The SHA in constants.py is still correct — it just wasn’t auto-updated in this PR.

Debugging with older images. A researcher needs to reproduce a bug that only occurs with last Tuesday’s image. They temporarily pin constants.py to the old SHA, run their debugging session, then revert.

CI commit-back failures. The pipeline that auto-commits constants.py occasionally fails — git conflicts, permission issues, transient GitHub API errors. When this happens, someone manually updates the SHA after verifying which build succeeded.

All three are recoverable because SHA tags are immutable. The old image still exists in the registry with its original tag. You can always go back.

The SHA-as-cache-key pattern

There’s a subtlety to SHA-based tagging that goes beyond “know which version is running.” The SHA also functions as a cache key for layer reuse.

Docker’s build cache works by layer identity. If layer N hasn’t changed, layers N+1 through M can reuse the cached versions. The SHA tag means:

• Same SHA → identical layers → identical CUDA kernels → guaranteed coherence
• Different SHA → need to check which layers changed → might still cache-hit on unchanged layers

The buildcache tag supports this. It’s a special tag that Docker BuildKit uses to store and retrieve layer caches from the registry. It mutates on every build, always pointing to the most recent layer cache. When a new build starts, BuildKit pulls buildcache to warm its local cache, then overwrites it with the new layers when done.

# In docker-bake.hcl
target "pretraining" {
  cache-from = ["type=registry,ref=inf5acr.azurecr.io/yolo:buildcache"]
  cache-to   = ["type=registry,ref=inf5acr.azurecr.io/yolo:buildcache,mode=max"]
}

This is latest done right. The buildcache tag mutates freely because it’s only used as a performance hint. If it’s stale or missing, the build is slower but still correct. No coherence constraint, no correctness risk.

The hierarchy of tag safety

Putting it together, there’s a clear hierarchy:

The rule is simple: training jobs use SHA tags (via constants.py). Developers who need a quick test use main. Nobody uses latest for anything that touches a GPU.

latest exists because Docker convention expects it and some tooling defaults to it. It’s a vestigial tag — present because removing it would break assumptions in tools we don’t control, serving no purpose in our actual workflow.

The lesson

“Don’t use latest” is common advice, but the why changes depending on your domain. For a web service, using latest means you might deploy the wrong version and get a 500 error. Annoying, rollback, done.

For ML training, using latest means you might deploy an image where the pre-compiled CUDA kernels don’t match the Python code calling them. The GPU dutifully executes the mismatched kernels. The gradients come out wrong. The loss curve looks plausible but slightly off. You train for two days before someone squints at a chart and says “that doesn’t look right.”

The fix is the same in both cases — use immutable tags. But the cost of getting it wrong is the difference between a five-minute rollback and a week of wasted GPU time. When each training run costs thousands of dollars in compute, “probably the right image” isn’t good enough.

Pin your tags. Verify your builds. Let latest be someone else’s problem.