Two Days to Merge Six Files: Docker CI Friction

I opened PR #21006 on Monday morning. Six files changed. A Dockerfile optimization that makes rebuilds 86% faster. The kind of PR you expect to merge by lunch.

It is now Wednesday. The PR is still open.

What the PR actually does

Our pretraining Docker image rebuilds its entire Python dependency layer every time uv.lock changes. I wrote about this in The uv.lock Cache Bust Problem — the short version is that one lockfile serves 20+ packages, changes ~4 times a day, and each change triggers a full reinstall. Eight to fifteen minutes of CI time, gone.

The fix: split uv sync into two phases.

Phase 1 installs from uv-stable.lock — a weekly snapshot of uv.lock. This layer stays cached for about a week because the file doesn’t change between snapshots.

Phase 2 runs uv sync --inexact with the real uv.lock. The --inexact flag means “don’t remove packages that are already installed but not in the lockfile.” So it only installs the diff — typically 0 to 5 packages.

# Phase 1: stable layer (cached ~1 week)
COPY ../../uv-stable.lock /app/yolo/uv-stable.lock
RUN uv sync --no-install-workspace --frozen --package mai_job

# Phase 2: differential install (seconds when diff is small)
COPY ../../uv.lock /app/yolo/uv.lock
RUN uv sync --inexact --no-install-workspace --frozen --package mai_job

A/B test results:

• Baseline rebuild: ~11 min average
• Optimized rebuild: ~1.5 min average
• Improvement: 86.5%

Six files. Clean optimization. Should be easy to merge, right?

The auto-commit feedback loop

The repo has a CI job called Commit & Update Tags. After containers are built, it pushes image reference updates back to the PR branch — new SHA digests for the freshly built images. This is fine in theory. In practice, it creates a cycle:

code change
  → CI runs (~70 min)
    → containers build
      → auto-commit pushes new image refs to branch
        → CI restarts (~70 min)
          → containers build (same images, nothing changed)
            → auto-commit (same refs, but it pushes anyway??)

The loop IS prevented from going infinite. There’s a label mechanism — image-exported plus ih-{hash}-exp — that tells the job “you already did this, stop.” So the second auto-commit doesn’t trigger a third run. But the damage is already done: the SECOND full CI run still happens. All the GPU tests, all the lint, all the type checking. Another 70 minutes of waiting for the exact same result.

My PR got hit with seven automated “Update container image references” commits over two days. Seven. Each one triggering varying amounts of re-validation.

Death by CI restarts

Here’s the timeline of what actually happened:

The “Update branch” button at the end was the one that really hurt. All checks were green. Approval was in. I hit the button because GitHub said the branch was behind main. It merged main into my branch, which changed the commit SHA, which restarted every single check.

The “Update branch” button is a trap.

Scope changes mid-flight

The original PR included optimizations for both the pretraining Dockerfile AND the rocket Dockerfile. Mid-review, I decided to scope down to pretraining-only. Simpler to review, easier to validate, less risk.

Smart decision for code quality. Terrible decision for CI efficiency. The first two CI runs had been building and testing both. Wasted cycles on work I then removed.

Lesson: decide your scope before you open the PR. Every scope change after opening is a tax.

The GitHub Checks state machine

During those two days of watching CI, I learned more about GitHub’s check system than I ever wanted to know. Here’s what nobody explains clearly.

The five states

Expected → Queued → In Progress → Success / Failed / Skipped

Expected is the confusing one. It means branch protection knows this check SHOULD exist — it’s in the required checks list — but the workflow hasn’t created the actual job yet. Like having a restaurant reservation but nobody’s walked in the door. The workflow file has been triggered, but the runner hasn’t picked up the job.

Queued means the job exists and is waiting for a runner to become available.

In Progress means it’s running.

The problem: GitHub’s summary line shows all of these at once, and they fluctuate constantly.

Why numbers go up AND down

You refresh the checks page and see:

6 in progress, 1 queued, 12 skipped, 5 expected, 37 successful

Refresh again:

8 in progress, 0 queued, 12 skipped, 3 expected, 38 successful

Refresh again:

4 in progress, 3 queued, 12 skipped, 0 expected, 42 successful

What’s happening:

• Jobs finishing: in_progress goes down, successful goes up. Normal.
• Jobs spawning children: expected goes down, queued/in_progress goes up. New jobs appear that didn’t exist before. The total count of checks INCREASES.
• New commit pushed: successful GOES DOWN. Previous run gets superseded, everything restarts from zero.

That third one is the gut punch. You’ve been watching numbers climb toward green for 50 minutes. Someone pushes a commit — or an auto-commit fires — and you’re back to 0 successful, 47 expected. The restaurant reservation analogy again, except this time they burned down the restaurant and rebuilt it.

The job tree

The full CI pipeline has a tree structure. Understanding this is the key to reading the checks page without losing your mind:

PR Required CI
├── lint/
│   ├── check-lock-file
│   ├── pre-commit-checks
│   ├── typechecker (pyright fast)
│   └── typecheck (pyright full)
├── cpu-tests/
│   ├── find_changed
│   ├── cpu_test_shard (data_forge, precook, etc.)
│   └── cpu_tests              ← GATE
├── gpu-tests/
│   ├── find_changed
│   ├── build_image            ← 14 min
│   ├── gpu_tests (0, 1, 2)   ← DON'T EXIST until build_image finishes
│   ├── redis_gpu_tests
│   └── gpu_tests_passed       ← GATE
├── rocket-gpu-tests/
│   ├── build_yolo_image
│   ├── run_yolo_gpu_tests
│   └── gpu_tests_rocket       ← GATE
└── build-containers/
    ├── Build Yolo Base Image
    ├── Build yolo image (amd64/arm64)
    ├── create_enroot_condor (condor1/condor2)
    ├── Verify All Builds Complete  ← GATE
    └── Commit & Update Tags        ← THE AUTO-COMMIT CULPRIT

Gate jobs are the ones that trip people up. cpu_tests, gpu_tests_passed, gpu_tests_rocket — these do zero work. They’re empty jobs whose only purpose is to wait for their siblings to finish and report a single pass/fail. GitHub branch protection requires the gate job, not the individual test shards.

Rule of thumb: if the job name ends in _passed or matches the group name exactly, it’s a gate job. It’ll go from Expected → Success in under a second once its dependencies clear.

Why jobs appear from nowhere

This one confused me for a while. You’re watching the checks page, total says 41 checks, and then suddenly it says 44. Three new jobs just… materialized.

Here’s why: gpu_tests (0), gpu_tests (1), gpu_tests (2) literally don’t exist as GitHub check runs until build_image finishes. The workflow uses a matrix strategy that only executes after the build step. So when build_image completes at minute 14, three new “In Progress” checks spawn into existence.

Same thing happens with rocket GPU tests, condor builds, and a few others. The CI isn’t broken — it’s just using late-binding job creation, and GitHub’s UI doesn’t distinguish between “this job hasn’t been created yet” and “this job failed to start.”

Estimating wait time

After two days of staring at this, I developed a mental model for predicting when a CI run will finish.

Step 1: Find the critical path — the longest chain of dependent jobs.

build_image (14 min) → gpu_tests shard 2 (53 min) → gpu_tests_passed (instant)
= ~67 min minimum

Step 2: Check if the critical path has already started. If build_image is already done, you’re looking at ~53 min from now. If it hasn’t started, add 14 minutes.

Step 3: Add 5-10 minutes of buffer for queue time, runner spin-up, and the occasional flaky test retry.

Step 4: Double it if you think anyone might push a commit.

The real formula:

actual_wait = critical_path × (1 + number_of_times_you_check_the_page)

That’s a joke. Mostly.

The emotional part

There’s a specific kind of frustration that comes from watching CI. It’s not the same as debugging a hard problem — at least with debugging you’re actively doing something. With CI, you’re just… waiting. Refreshing. Watching numbers.

You develop superstitions. “If I stop looking at it, it’ll go faster.” You learn the exact refresh interval where the numbers change (about 30 seconds on GitHub). You start recognizing individual job names and their typical durations like old friends. “Oh, cpu_test_shard (data_forge) is done already, that’s fast today.”

And then the auto-commit fires and everything resets and you make a noise that your coworkers pretend not to hear.

The worst part is the math. The PR itself — the actual code changes — took maybe 4 hours of focused work including the A/B testing. The CI overhead was 20+ hours of pipeline time and 2 days of calendar time. The ratio of “time spent writing code” to “time spent waiting for permission to ship code” was roughly 1:5.

For six files.

What I’d do differently

1. Don’t merge main into your PR branch. Each merge restarts CI. If you need to sync, batch it — wait until you have other changes to push at the same time. One restart is better than three.

2. Scope the PR before opening it. Changing scope mid-PR wastes every CI run that happened before the scope change. Decide what’s in and what’s out, then open.

3. The auto-commit cycle is a known tax. For Dockerfile-only changes, the built image contents might not change, so the auto-commit is pushing identical refs. Masatoshi’s advice: consider skipping the auto-commit for PRs that only modify build config. We don’t have this automation yet, but we should.

4. Learn the state machine. Understanding Expected vs Queued vs In Progress saves you from refreshing every 30 seconds wondering why your check count went down. It didn’t go down — jobs just moved between states.

5. Don’t push after approval. If a reviewer catches something, yes, fix it. But know that you’re buying another ~70 minutes. If it’s cosmetic, ask if it can be a follow-up.

6. The “Update branch” button is a trap. It merges main into your branch, changes the commit SHA, and restarts all checks. If your checks are green and you just need to update, consider rebasing locally and force-pushing — at least then you control when the restart happens.

7. Auto-merge exists. If checks are green and approval is in, enable auto-merge and walk away. Let the machine handle the timing instead of you hitting refresh like a lab rat pressing a lever.

The bigger picture

This isn’t a story about one bad PR. It’s about the cumulative cost of CI complexity in a large monorepo. Every safeguard — branch protection, required checks, auto-commit for image refs, gate jobs — exists for a good reason. Each one prevents a real class of bugs. But stacked together, they create a system where merging six files takes two days.

The DevEx team (my team) builds the CI. We’re the ones who added these safeguards. And we’re the ones who suffer the most when they interact badly, because our PRs tend to touch the build system itself — which triggers everything.

There’s no clean fix. You can’t just remove safeguards. But you can make the feedback loop faster (which is what this PR does — 86% faster rebuilds), and you can teach people how to read the system so they spend less time confused and more time productive.

Or you can write a 2000-word blog post about it at 2 AM while waiting for your seventh CI run to finish.

That works too.