The 30-Second Fix Nobody Did for 3 Months: Stale Docker Prepuller

2026/03/09

BUILDcidockergithub-actionskubernetes

The yolo monorepo builds a ~30GB Docker image on every merge to main. That image gets pushed to Falcon ACR. On every GPU node in the cluster, a DaemonSet pre-pulls that image so when someone launches a training job, it’s already there. No 45-minute cold start staring at ImagePullBackOff.

The prepuller config had been pointing at a 3-month-old image tag. Since November 2025. The image builds every day. Nobody updated the tag.

Because updating the tag means opening a PR to a completely different repo. And nobody remembers to do that.

The setup

Two repos. yolo is the monorepo — code, models, CI, the Docker image build. mango is the infra repo — Kubernetes configs, ArgoCD apps, the prepuller DaemonSet.

The prepuller lives in mango as a DaemonSet with an images.yaml that lists which images to pull. One of those entries looks like this:

- name: pull-yolo-pretraining-6305057
  image: maifalcon.azurecr.io/yolo/pretraining:6305057
  command: ["/bin/sh", "-c", "echo 'Pulled yolo-pretrainig-6305057'"]

6305057 is a git SHA from November. There’s also a typo — pretrainig — that’s been there the whole time. Nobody noticed because nobody looks at this file.

ArgoCD watches mango’s main branch. When images.yaml changes, it rolls out the new DaemonSet config. The mechanism works fine. The problem is upstream: nothing triggers the change.

How this surfaced

Three separate people noticed independently within the same week.

Yiran from HPC flagged it in a DM: “I think if possible, we should also update the image tag(s) here for prepulling.” Ajinkya from Falcon opened a thread asking if they could use stable tags like latest so the prepuller would always get the newest image.

Yiran shut that down immediately: “Stable tag is probably not doable… it normally causes confusion down the line. Cannot debug which image latest was.”

Which is exactly right. When a training job breaks and you’re debugging what image it’s running, you need to know which exact build. latest tells you nothing. Immutable tags are non-negotiable.

Then Yiran said the magic words: “CI folks can probably just update this line here as images are built from main.”

That’s me. I’m CI folks.

Why not stable tags

This came up enough in the thread that it’s worth spelling out.

Stable tags (latest, stable, nightly) feel like the obvious answer. Point the prepuller at latest, never update the config again. But:

Immutable tags — where the tag changes every time the image changes — solve all three. The prepuller spec changes, Kubernetes rolls out new pods, nodes pull the new image, and you always know exactly what’s running.

Content-hash tags

Yolo already generates content-hash tags: yolo-${SHA256_HASH}. The hash is derived from the image content, not the git SHA. This matters.

If you use git SHAs as tags, every merge to main produces a new tag even if the Docker image didn’t change at all. Maybe the merge only touched Python tests, or docs, or some unrelated package. The image is identical, but the tag is different, so the prepuller restarts, every node re-pulls 30GB, and the DaemonSet cycles — all for nothing.

Content-hash tags only change when the image content actually changes. Same image, same hash, same tag. The prepuller config doesn’t change, ArgoCD doesn’t sync, nodes don’t re-pull. Updates only happen when there’s actually a new image to pull.

This is the difference between “update on every CI run” and “update when it matters.”

The cross-repo problem

The actual fix is trivial — change three strings in a YAML file. The mechanism to make it happen automatically across repo boundaries is the interesting part.

Repo A builds an image. Repo B needs to know about the new tag. GitHub Actions doesn’t have native cross-repo event wiring. You can’t just on: push from a different repo.

The answer is repository_dispatch. Repo A sends an HTTP event to Repo B’s API. Repo B has a workflow that listens for that event type. It’s GitHub’s version of a webhook between repos.

But you need auth. Repo A needs a token that can write to Repo B. A personal access token would work but it’s tied to a person — they leave, it breaks. A fine-grained token scoped to one repo would work but it needs manual rotation.

We already had the right tool: MAI_DEPLOYMENTS_WRITER — a GitHub App installed on both repos. Mango’s existing bump-container-image.yml already uses it. No new secrets, no new tokens, no new approvals. Just reuse what’s there.

Finding it was just grep. Search mango for MAI_DEPLOYMENTS_WRITER and you land on bump-container-image.yml — a reusable workflow written by Yogesh Garg. It uses tibdex/github-app-token@v2 with MAI_DEPLOYMENTS_WRITER_APP_ID + MAI_DEPLOYMENTS_WRITER_PRIVATE_KEY to generate a token, then runs kustomize edit set image to update image tags in kustomization overlays and commits directly to main. Our prepuller automation literally copies this pattern: generate app token, checkout with it, modify file, commit, push to main. No invention needed — just reuse.

The two PRs

Yolo side (PR #24045) — a new job in the existing image build workflow:

notify_mango_prepuller:
  needs: [verify_all_builds]
  if: github.ref == 'refs/heads/main'
  steps:
    - uses: peter-evans/repository-dispatch@v3
      with:
        token: ${{ steps.app-token.outputs.token }}
        repository: infinity-microsoft/mango
        event-type: yolo-image-updated
        client-payload: >-
          {
            "git_sha": "${{ github.sha }}",
            "yolo_hash": "${{ needs.verify_all_builds.outputs.yolo_hash }}",
            "tag_hash": "yolo-${{ needs.verify_all_builds.outputs.yolo_hash }}",
            "source_run_id": "${{ github.run_id }}"
          }

Only fires on main, only after all builds succeed. The payload carries everything mango needs: the content-hash tag, the git SHA for traceability, and the source run ID so you can link back to the build that triggered it.

Mango side (PR #10911) — a new workflow that receives the dispatch:

on:
  repository_dispatch:
    types: [yolo-image-updated]
  workflow_dispatch:
    inputs:
      tag_hash:
        description: 'Image tag (e.g., yolo-abc123def)'
        required: true

The workflow_dispatch input is there for manual recovery — if the automation breaks or you need to force a specific tag, you can trigger it by hand.

A Python script parses images.yaml, finds the yolo pretraining entry, and updates three things: the container name suffix, the image tag, and the echo message. Then commits directly to main.

ArgoCD picks it up from there.

What the automation does NOT do

Worth being explicit:

Verifying it works

The nice thing about this design is you can test each half independently.

  1. Merge the mango PR first. The new workflow exists but nothing triggers it yet.
  2. Run it manually with workflow_dispatch and a known tag. Watch it update images.yaml, verify the commit looks right, verify ArgoCD syncs.
  3. Then merge the yolo PR. Next image build on main sends the dispatch. Watch the mango workflow trigger. End-to-end.

If the dispatch fails (auth issue, mango workflow error), the yolo build still succeeds — the notification job is independent. You’ll see a failed job in the Actions UI but nothing is blocked.

The actual lesson

This entire project exists because a human had to remember to do something after a CI event, and humans don’t do that. They don’t do it because it’s boring, because it’s in a different repo, because the build still “works” without it, because there’s no alert when it’s stale. The tag was 3 months old and nothing was on fire.

That’s the most dangerous kind of staleness — the kind where nothing breaks, so nobody notices, and then one day someone’s training job cold-starts for 45 minutes and they go digging.

If a task is “after X happens in repo A, update Y in repo B” — automate it. The mechanism barely matters. repository_dispatch, a cron job, a webhook, a carrier pigeon with a USB drive. Whatever. Just don’t make a human remember.