Docker Builds: How to Prove an Optimization Actually Worked

I wrote a clever Docker optimization. Two-phase uv sync, stable lockfile trick, the whole thing. Felt great. Wrote a blog post about it. Then someone asked the obvious question: “Does it actually work?”

And I realized I had no idea.

Update: I now have an idea. Scroll to the verdict if you’re impatient, or straight to the full data report if you want all 12 CI runs worth of evidence.

The problem with “just benchmark it”

Normal software optimization has a nice workflow. Run the thing. Time it. Change the code. Time it again. Compare numbers. Celebrate or cry.

Docker image builds for ML training are not normal software. Here’s what “just run it and see” looks like for us:

• GPU runners — the build compiles CUDA kernels. My laptop has an M-series chip. It will not be participating.
• Private registries — base images live in Azure Container Registry behind auth. You need the right credentials AND Tailscale VPN for our internal PyPI.
• Beefy machines — CI runners have 32 cores, 128GB RAM, 1.2TB disk. My laptop has opinions about compiling Flash Attention 3.
• docker buildx bake — we don’t use plain docker build. It’s an HCL bake file orchestrating multiple images. Good luck reproducing that in your kitchen.
• Scheduled builds — production builds run every 2 hours on main via GitHub Actions cron. That’s the baseline you’re comparing against.

So “run it 5 times and average” doesn’t work. You can’t run it at all. Your laboratory is a CI pipeline, your measurements come from GitHub Actions logs, and your sample size is “however many builds happened to run this week.”

This is less “engineering benchmarking” and more “building a court case with whatever surveillance footage exists.”

The three scenarios (and why only one matters)

The optimization splits uv sync into two phases: Phase 1 installs from a stable lockfile that barely changes, Phase 2 handles the diff. Sounds great in theory. But to prove it works, you have to think about when it matters.

Scenario A: Steady state. Nothing changed. No dependency updates. Both the old 1-phase approach and the new 2-phase approach cache everything perfectly. Expected result: same duration. This is your “first, do no harm” test. Important but boring.

Scenario B: Minor dep bump. Someone added requests>=2.32 or bumped a patch version. uv.lock changed, uv-stable.lock didn’t. Old approach: cache bust, full 20-minute reinstall of every single package. New approach: Phase 1 cached (stable lock unchanged), Phase 2 installs only the diff — maybe 30 seconds. This is the scenario that proves the optimization.

Scenario C: Major dep overhaul. Someone upgraded PyTorch or did a lockfile refresh. Both lockfiles changed. Both approaches do a full install. New approach is slightly slower because it runs uv sync twice. Expected result: roughly the same, maybe a few percent worse.

Here’s the kicker: if you just compare a bunch of CI runs without thinking about which scenario they represent, you’ll mostly see Scenario A — because most builds don’t change deps. You’ll compare two approaches that both cache perfectly and conclude “no difference.” Which is technically true and completely useless. Like comparing two cars’ fuel efficiency while they’re both parked.

You have to deliberately trigger Scenario B. Make a trivial dep change on your PR branch, run the build, and compare against baseline. Science requires controlled experiments, even when your lab is a YAML file.

Building a proper test lab (in YAML, obviously)

After fumbling around with gh run list and manually timing things from production build logs, I gave up and did the right thing: I built a dedicated A/B test workflow.

It’s called test-pretraining-build.yml. One click of workflow_dispatch, full controlled experiment. Here’s the design:

Two parallel jobs, identical runners (ubuntu-2404-x64-32c.128g.1200g):

Job 1 — “Baseline (main)”:

• Checks out main branch (old single-phase Dockerfile)
• Pass 0: Warm build — populate the buildx cache, push to registry. This primes the pump.
• Then 3 rebuild passes. Before each one, append a comment to uv.lock (echo "# cache-bust-N" >> uv.lock) to simulate a dependency change.
• Time each pass with date +%s before and after docker buildx bake.

Job 2 — “Optimized (this branch)”:

• Checks out the PR branch (new 2-phase Dockerfile)
• Same warm build, same 3 cache-bust rebuilds
• The key: uv-stable.lock is NOT touched, so Phase 1 should stay cached

Job 3 — “Compare Rebuild Times”:

• Runs after both complete
• Computes min/max/avg across the 3 rebuild passes
• Outputs a markdown comparison table to GitHub Step Summary

Why this is better than scraping production logs:

• Both variants run simultaneously on the same runner type — no time-of-day variance, no “the network was slow on Tuesday” excuses
• The warm build (Pass 0) primes the cache so rebuilds measure cache effectiveness, not cold start speed
• 3 rebuild passes give min/max/avg — enough to see if results are noisy
• Self-contained: one button click = full comparison with a results table. No archaeology required.

The workflow itself is about 200 lines of YAML. Which means I spent more lines proving the optimization works than I spent writing the optimization. This is normal. This is fine. I’m fine.

The verdict

From the successful A/B run (Run ID 22248141298, Feb 21 2026):

Average rebuild: 13m 03s → 6m 07s. That’s 53% faster.

The variance is low on both sides — within ~40 seconds across all 3 passes. This isn’t statistical noise hoping to be signal. It’s a consistent, reproducible, real improvement.

The warm builds are roughly the same (within 90 seconds), which confirms Scenario A: when nothing changes, the optimization doesn’t make things worse. First, do no harm. Check.

But one run with 3 passes isn’t exactly a doctoral thesis. I ran a lot more experiments — 12 CI runs across 4 configurations, Gen 3 workflow with 3 trials × 3 passes each. The full data report is below, and the headline holds up: the optimization is real, but the story gets more interesting when you add push.

Poking holes in my own case

A good defense attorney pokes holes in their own case before the opposing counsel gets to. Here’s me, cross-examining myself.

Now the actually interesting concern:

The cache-bust is fake. Appending # cache-bust-1 to uv.lock makes Docker see a changed file and bust the layer cache — mission accomplished. But uv sync --frozen reads the lockfile as-is, so the comment doesn’t change what gets installed. Both sides reinstall the same packages.

Is that unfair? No, and here’s why: we’re testing Docker’s layer cache behavior, not uv’s dependency resolver. The question isn’t “can uv install a new package?” — it’s “when the lockfile changes, how much work does Docker redo?” A real dep bump would actually make the optimized approach look better, because Phase 2 would only install the new package while the baseline still reinstalls everything.

The one scenario that could flatter us: If uv-stable.lock was just refreshed, Phase 1 and Phase 2 install nearly identical packages. The test is most representative when uv-stable.lock is a few days old — which is the normal state, because we only refresh it weekly. So the test reflects reality. Good.

I’m satisfied. The opposing counsel can sit down.

Wait, why did I remove –push?

After the trial concluded, I kept thinking about details. This is either thoroughness or anxiety. Let’s call it thoroughness.

First up: the --push flag. My test workflow originally didn’t push images to the registry. Makes sense, right? We’re measuring build time, not upload speed. Why pollute ACR with garbage test images?

But then the little voice kicked in: In production, builds push. That’s part of the real pipeline. If you’re trying to prove the optimization works under production conditions, shouldn’t the test match production?

Counter-argument from the other little voice: But the push phase is the same for both approaches. It’s a constant. Removing it just isolates the build phase more cleanly.

Both little voices had a point. So I stopped arguing with myself and did the engineer thing — made it configurable:

# Build only (isolates build time)
gh workflow run test-pretraining-build.yml -f push=false

# Build + push (matches production)
gh workflow run test-pretraining-build.yml -f push=true

Ran 3 trials of each in parallel. Same wall time, double the data. If the delta is similar with and without push, we know the optimization lives in the build phase, not the push phase. Which is what we’d expect — but “expecting” and “proving” are different things. That’s literally the thesis of this entire blog post.

The lesson: When you’re not sure whether a variable matters, don’t argue about it. Test both. Especially when testing both costs the same as testing one.

The Docker jargon that tripped me up

I got confused by my own blog post. Specifically, I wrote “same builder” and “different runners” and then couldn’t remember what those words meant at 2am. Here’s the glossary I wish I’d had.

Runner vs Builder vs Job:

• Runner = a physical/virtual machine. Each GitHub Actions job gets its own runner. 1 job = 1 runner = 1 machine. When the job finishes, the runner is destroyed. Gone. Ashes to ashes, YAML to YAML.
• Builder = a Docker buildx process inside the runner. Created by docker/setup-buildx-action. Manages the layer cache. Multiple build passes within the same job use the same builder — that’s how cache persists between Pass 0, 1, 2, 3.
• Job = a unit of work in a GitHub Actions workflow. Separate jobs run on separate runners, which means separate builders. The layer cache from Job 1 cannot leak into Job 2. They’re on different machines.

So when the test workflow runs baseline and optimized as two separate jobs, they’re on two different machines with two different builders. No cache contamination. The courtroom has two separate evidence rooms.

“Within-runner variance” vs “cross-runner variance”:

• Within-runner: Pass 1, 2, 3 on the same machine. The ~40s spread we see. Just normal noise — CPU thermals, background processes, disk I/O jitter. The kind of variance you’d see timing anything 3 times on the same computer.
• Cross-runner: Baseline job lands on Machine A, optimized job lands on Machine B. Same spec (same CPU/RAM/disk), but different physical hardware. Might perform slightly differently. This is the harder-to-control variable.

Why cross-runner variance is fine here: Our signal is 53% — that’s 6+ minutes of difference. Hardware variance between same-spec runners is maybe 10-15%. You don’t need a microscope when the difference is visible from space.

But in production, each build is on a different runner

This was the question that actually kept me up at night. Everything above is about the test. But production is different.

The test uses local buildx cache — the same builder persists across passes within a job. But production builds run every 2 hours on different runners. Different runners = different builders = no local cache. So how does the optimization work in production?

Answer: registry cache. The buildx bake config has:

cache-from = type=registry,ref=<acr>/cache
cache-to   = type=registry,ref=<acr>/cache

Every production build pulls cached layers from ACR at the start and pushes updated layers back when done. The cache doesn’t live on the runner — it lives in the registry. Different runners, different builders, doesn’t matter. The cache survives because it’s not on the machine that dies.

The test actually simulates this correctly: Pass 0 pulls cache-from=type=registry to warm the local builder, then Passes 1-3 use that local cache. In production, the cache-from=type=registry pull happens every build. The cache hit/miss logic is identical — Docker checks “did the files in this COPY layer change?” Whether the cached layer comes from local storage or the registry doesn’t change the answer.

The optimization works in production because: When uv.lock changes but uv-stable.lock doesn’t, the registry cache still has the Phase 1 layer. Stable lock unchanged → cache hit from registry → skip Phase 1 entirely. Only Phase 2 rebuilds.

This is the insight that made the whole thing click. The test proves the mechanism works. The registry cache is what makes the mechanism portable across runners. Two pieces, same puzzle.

One more thing: workflow_dispatch only works if GitHub knows about it

Bonus debugging story. Because every project needs a “I lost 45 minutes to something stupid” section.

I tried creating a separate workflow file (test-pretraining-build-with-push.yml) for the push variant. Pushed it to the branch. Tried to trigger it:

HTTP 422: Invalid Argument - workflow test-pretraining-build-with-push.yml not found

Excuse me? It’s right there. In the repo. On the branch I just pushed. ls can see it. cat can read it. GitHub Actions cannot.

Turns out GitHub needs to have seen the workflow file on some branch’s history before you can dispatch it via API. The original test-pretraining-build.yml worked because it existed on a previous branch. The new file was brand new — GitHub’s internal workflow registry had never indexed it. It’s like trying to call a phone number that hasn’t been activated yet.

Fix: Merged both variants into one workflow with a push input flag. One registered workflow, two modes. Problem solved, 45 minutes I’ll never get back, lesson permanently learned.

(If you’re keeping score: that’s yet another piece of “proving infrastructure” that took longer than the actual optimization. The ratio is now approximately 80x. I’m fine.)

The two-tool strategy

We ended up with two complementary tools:

test-pretraining-build.yml — the dedicated A/B test. Use it for PR evidence. “Here’s proof it works.” Run it 2-3 times, paste the results table in your PR description, watch reviewers nod instead of squinting.

compare-build-duration.py — a script that pulls timing from production builds via gh CLI. Use it for post-merge monitoring. “Is it still working after 2 weeks?” Compares scheduled main builds over time, spots regressions before anyone notices.

One proves the optimization works. The other proves it keeps working. Belt and suspenders, but for Docker builds.

The full data report (Gen 3, multi-trial)

One run with 3 rebuild passes was enough to be convincing. But “convincing” and “bulletproof” are different words. So I built a Gen 3 version of the test workflow — matrix-based, 3 independent trials per run, each trial doing its own warm build + 3 cache-bust rebuilds. That’s 9 rebuild datapoints per run. Then I ran it across 4 configurations: pretraining and rocket_yolo images, with and without --push.

12 CI runs. Roughly 100 rebuild measurements. Here’s everything.

Configuration 1: Pretraining, no push

This is the cleanest test — isolates build time, no push overhead. All 3 runs completed with full 3×3 data (3 trials × 3 rebuilds each = 9 datapoints per side).

Run 22367207173

Aggregate: Mean baseline 3m 13s → Mean optimized 1m 24s. Δ 1m 49s (56.3% faster). Stddev: baseline 1m 00s, optimized 1m 01s. N=9 each.

Run 22367214374

Aggregate: Mean baseline 3m 03s → Mean optimized 1m 20s. Δ 1m 43s (56.2% faster). N=9 each.

Run 22367222506

Aggregate: Mean baseline 2m 49s → Mean optimized 1m 50s. Δ 58s (34.6% faster). N=9 each.

Pretraining no-push summary: Across 3 runs (27 rebuild datapoints per side), the optimized build is consistently faster. The percentages vary — 34% to 56% — because the baseline itself varies a lot (standard deviation of ~1 minute). The optimized side tends to converge to sub-minute times on the 2nd and 3rd pass within a trial. That’s the stable lockfile cache hitting.

But here’s the catch: no-push is a useful isolation test — it proves the Docker layer cache mechanism works. But it understates the real-world impact. Those 35-56% improvements sound solid until you realize we’re shaving ~1-2 minutes off a 2-3 minute baseline. Nice, not transformative.

Production always pushes. There is no optional flag. Line 298 of build_yolo_base_and_rocket_containers.yml is just --push. And push cost is proportional to what changed in the layer. When the optimization keeps Phase 1 cached, the Phase 1 layer already exists in ACR from the previous build. Docker says “blob already exists, skip.” Only the small Phase 2 diff layer needs uploading — and that’s nearly instant. Meanwhile, baseline’s cache-busted Phase 1 produces a fat new layer that has to be pushed in its entirety. That’s why optimized push adds ~0 time while baseline push adds ~8 minutes.

No-push proves the mechanism. Push=true is where the mechanism pays rent.

Configuration 2: Pretraining, push=true

Now add --push to the build — push the image to ACR after building. This matches production. Push is the same work for both variants, so it should add a constant to both sides. The optimization delta should survive. That’s the hypothesis. Let’s see.

Three partial runs (some optimized trials failed — flaky ACR auth, classic), plus one full 3×3 success.

Actually, “flaky ACR auth” is a lazy explanation. I kept digging. The 3 partial runs (22367569194, 22367579338, 22367588906) each had 2 out of 3 optimized trials fail with the same error:

failed to push inf5acr.azurecr.io/yolo/pretraining:latest: 416 Requested Range Not Satisfiable

Always optimized trials. Never baseline. Which is suspicious in exactly the right way.

Here’s what happened: I triggered all 3 workflow runs simultaneously. Each has 3 baseline + 3 optimized = 6 trials, so 18 jobs are hitting ACR. The optimized builds finish faster — that’s the whole point of the optimization — so their warm builds all try to --push to the same ACR tag at roughly the same time. ACR chokes on concurrent blob uploads to the same tag. 416 Requested Range Not Satisfiable is the registry saying “someone else is already writing this blob, go away.”

The irony is beautiful: the optimization fails because it’s too fast. A thundering herd on the registry, caused by efficiency. Meanwhile, baseline’s warm builds take longer and naturally stagger themselves. Slow enough to avoid the stampede.

The 4th run (22369931206) succeeded because I triggered it alone, an hour later. No contention. One at a time, like civilized builds.

Run 22367569194 — 3 baseline × 1 optimized (2 opt trials failed)

Aggregate: Mean baseline 11m 13s → Mean optimized 1m 37s. Δ 9m 35s (85.5% faster). N=9 baseline, 3 optimized.

Run 22367579338 — 3 baseline × 1 optimized

Aggregate: Mean baseline 11m 42s → Mean optimized 1m 28s. Δ 10m 14s (87.5% faster). N=9 baseline, 3 optimized.

Run 22367588906 — 3 baseline × 1 optimized

Aggregate: Mean baseline 11m 18s → Mean optimized 1m 29s. Δ 9m 49s (86.9% faster). N=9 baseline, 3 optimized.

⭐ Run 22369931206 — Full 3×3 success. The star witness.

Aggregate: Mean baseline 11m 14s → Mean optimized 1m 30s. Δ 9m 43s (86.5% faster). Stddev: baseline 1m 04s, optimized 4 seconds. N=9 each.

That optimized standard deviation. Four seconds. Across 9 independent rebuild measurements. The stable lockfile cache doesn’t just make things faster — it makes them boringly predictable. Every rebuild lands at ~90 seconds, plus or minus the time it takes to sneeze.

Pretraining push summary: 85–88% faster on rebuilds with push. The baseline jumps from ~3 minutes (no push) to ~11 minutes (with push) because pushing layers to ACR takes ~8 minutes. The optimized side? Still ~1.5 minutes. The push is nearly instant because the only changed layers are tiny. The optimization makes the push cheaper too, not just the build.

Configuration 3: Rocket_yolo, no push

Rocket_yolo is a smaller image — fewer dependencies, faster builds. The optimization should still help, but the absolute numbers are smaller, so the delta will be smaller too. All 3 runs: full 3×3 success.

Run 22368941352

Aggregate: Mean baseline 1m 45s → Mean optimized 51s. Δ 53s (51.1% faster). N=9 each.

Run 22368947754

Aggregate: Mean baseline 2m 14s → Mean optimized 51s. Δ 1m 22s (61.5% faster). N=9 each.

Run 22368953939

Aggregate: Mean baseline 1m 50s → Mean optimized 53s. Δ 56s (51.5% faster). N=9 each.

Rocket no-push summary: 51–62% faster. The optimized side converges to ~51 seconds — essentially just the uv sync no-op check. The baseline bounces around more because when its single-phase cache busts, it reinstalls everything, and “everything” for rocket_yolo is variable depending on what’s cached from the warm build.

Configuration 4: Rocket_yolo, push=true

And here’s where the story gets interesting. Or rather, where the story stops being interesting — which is itself interesting.

Run 22368959020 — Full 3×3

Aggregate: Mean baseline 10m 53s → Mean optimized 9m 13s. Δ 1m 40s (15.3% faster). N=9 each.

Run 22368965603 — 1 baseline × 2 optimized

Aggregate: Mean baseline 10m 09s → Mean optimized 9m 33s. Δ 36s (6.0% faster). N=3 baseline, 6 optimized.

Run 22368971930 — 2 baseline × 3 optimized

Aggregate: Mean baseline 9m 46s → Mean optimized 9m 50s. Δ 4s SLOWER (0.8% regression). N=6 baseline, 9 optimized.

Rocket push summary: The optimization barely moves the needle — 15% at best, and one run actually shows a slight regression. Why? Because push dominates. Rocket_yolo’s build phase is fast to begin with (~1-2 minutes), but pushing the image to ACR takes ~9 minutes regardless. When 90% of the wall clock is push, a 50% improvement on the remaining 10% is… math… 5%. That’s inside the noise.

This is actually a useful result. It tells you where to look for the next optimization. Hint: it’s the push, not the build.

But I kept digging, and this one turned out to be the most interesting finding in the whole analysis.

Why Rocket doesn’t get push savings (structural mismatch)

The “push dominates” explanation is correct but incomplete. It’s Amdahl’s Law — you can’t speed up the whole system by optimizing the part that isn’t the bottleneck. But why doesn’t the optimization reduce push time for rocket the way it does for pretraining? That’s the deeper question.

For pretraining+push, optimized rebuilds take ~90 seconds total — build AND push. The push is nearly instant because the stable lockfile layer already exists in ACR. Docker says “blob already exists,” skips the upload. Only the tiny Phase 2 diff layer needs pushing. That’s the whole payoff.

Rocket doesn’t get this payoff. And it’s because of two structural differences in how the Dockerfiles are written.

Problem 1: --mount=type=bind vs COPY

Pretraining uses COPY:

COPY ../../pyproject-stable.toml /app/yolo/pyproject.toml
COPY ../../uv-stable.lock /app/yolo/uv.lock
RUN uv sync ...

Rocket uses bind mounts:

RUN --mount=type=bind,source=uv-stable.lock,target=uv.lock \
    uv sync ...

COPY bakes the file content hash into the layer key. When ACR already has that layer, push skips it entirely — content-addressed deduplication at the registry level. Bind mounts don’t create content-addressed layers the same way. The RUN layer’s cache key depends on the mount content, but the resulting layer isn’t deduplicated against the registry the same way a COPY-then-RUN sequence is.

Problem 2: The source=. bind mount

This is the real killer. Rocket has a third step that mounts the entire repo:

RUN --mount=type=bind,source=.,target=/app/yolo,rw \
    uv sync --package rocket --frozen

Any file change in the repo — including our uv.lock cache-bust — invalidates this layer. It rebuilds and produces a new large layer that must be pushed. Both baseline and optimized hit this identically. The push cost is the same for both, and push is the bottleneck (~9 minutes).

So the 2-phase optimization saves build time (~50% in no-push) but doesn’t save push time. The optimization was designed for pretraining’s COPY-based Dockerfile architecture. Rocket’s bind-mount approach doesn’t get the registry-level deduplication that makes push nearly free.

This isn’t a failure of the optimization — it’s a structural mismatch. And it tells you exactly what you’d need to change to make rocket benefit: refactor toward COPY-based layering so the registry can deduplicate stable layers. Different architecture, different fix needed.

One more thing about the no-push “improvement” for rocket.

Look at the raw rebuild seconds across all 3 no-push runs:

Baseline:  [172, 52, 52, 171, 168, 52, 179, 52, 52, ...]
Optimized: [52,  51, 51, 52,  51,  51, 53,  51, 53, ...]

Baseline is bimodal — the first rebuild in each trial hits ~170 seconds, then subsequent passes drop to ~52 seconds. The local buildx cache warms up within the same builder, so even baseline’s cache-busted layer resolves fast after the first hit. Optimized is flat at ~52 seconds from the start.

The “51% improvement” is mostly driven by those first-pass outliers. By pass 2-3, both variants converge to identical times. The headline percentage is real, but it’s measuring the first-rebuild penalty, not a persistent advantage. In production — where each build runs on a fresh runner with no local cache carryover — the first-rebuild penalty is the only one that matters. But that’s still just ~2 minutes saved on a ~9-minute push. Small potatoes compared to pretraining’s 9+ minute savings.

The grand summary

The pretraining+push row is the production-representative one, and it’s not close. 86% faster, consistent across 4 independent runs, sub-minute standard deviation on the optimized side. The stable lockfile cache means rebuilds after a dep bump take 90 seconds instead of 11+ minutes. On a build that runs every 2 hours, that’s a lot of saved CI time.

The rocket+push row is the honest row. When push dominates, the build optimization can’t save you. Different bottleneck, different solution needed. Proving where the optimization doesn’t help is just as valuable as proving where it does — it tells you what to work on next.

The punchline

Writing the optimization took about a day. It’s maybe 6 lines of meaningful Dockerfile changes and a script to generate uv-stable.lock.

Proving it works? The dedicated test workflow is ~200 lines of YAML. The comparison script is another ~200 lines of Python. The fairness analysis required me to inventory every cache layer in the build pipeline and argue with myself about each one.

The proof infrastructure is 60x larger than the change it’s proving.

But here’s the thing — the jury came back. 86% faster rebuilds on pretraining with push — the production-representative config. Not from one run with 3 passes, but from 12 runs across 4 configurations, with enough datapoints to make a statistician nod approvingly. And the one config where it didn’t help told us exactly where to look next.

And someone will ask. In six months, when a regression shows up and someone’s staring at build times wondering what went wrong. They’ll find the test workflow, hit workflow_dispatch, and get a fresh comparison in 40 minutes. No archaeology, no guesswork, no “seems fine I guess.”

That’s the real optimization. Not the Dockerfile trick — the ability to prove the Dockerfile trick works, on demand, forever.

Build the court case. The code is the easy part.