Docker Builds: Shipping the uv-stable.lock Optimization

I have a PR that should make our Docker builds faster. I’ve had it for four days. I can’t figure out how to prove it works.

This is the third post in a series. Part 1 identified the problem: uv.lock changes ~4 times a day and busts the Docker layer cache every time, costing 8-15 minutes per rebuild. Part 2 tested the fix: a two-phase Dockerfile with a stable lockfile snapshot. We found ~60% speedup — mostly in registry push time, not install time. Clean experiment, clear signal.

Now I need to ship it. And shipping turns out to be the hard part.

The validation attempt

PR #19942 has the new Dockerfile. To validate it, I triggered 3 builds on the PR branch and compared timing against recent main builds.

New Dockerfile (PR branch):

Old Dockerfile (main branch, last 6 runs):

Stare at those numbers for a minute. What did we learn?

On cache misses, main takes 28 minutes. The PR branch takes 21-27 minutes. That’s… maybe faster? On cache hits, main takes 16-18 minutes. The PR branch never got a clean cache hit during my test runs.

One run hit 55 minutes because of runner contention — nothing to do with the optimization, just a machine that was too busy to finish in time. Two runs failed for unrelated reasons. I have maybe 3 usable data points.

This is not a compelling dataset.

The testing problem

Here’s what I’ve slowly realized: testing a cache optimization in CI is like testing an umbrella by checking if it rained. You can hold the umbrella up all day, but whether it “works” depends entirely on weather conditions you don’t control.

In our controlled experiment (Part 2), we could set up exact conditions: warm the cache, simulate a lockfile change, rebuild. Clean A/B comparison. But that was a test harness running on a dedicated workflow. The real question is: does this help in production CI, where builds happen on whatever runner is available, with whatever cache state that runner happens to have?

And I can’t answer that question before shipping it. The only way to know if it helps in production is to put it in production.

This feels like a fundamental category problem. Some optimizations you can test in isolation — “this algorithm is O(n) instead of O(n^2), here’s the benchmark.” Cache optimizations aren’t like that. The performance depends on the environment’s memory, which is exactly the thing you can’t reproduce in a test.

Same-runner bias

There’s a more specific version of this problem that’s been bugging me.

When I run builds on the PR branch, they execute on the same runner pool as main. The runners have shared caches. So my PR branch might be benefiting from cache layers that main built earlier — or vice versa. The measurements aren’t independent.

It’s like testing two diets by having the same person eat breakfast from Diet A and lunch from Diet B. The lunch results are contaminated by breakfast.

In a proper experiment, you’d want isolated runners — one pool for old Dockerfile, one for new, no shared cache. We don’t have that. We have a shared pool of ephemeral runners, and builds land on whichever one is free.

Even worse: the PR branch doesn’t have the uv-stable.lock refresh workflow running (because it hasn’t merged yet). So the stable lockfile in my PR is frozen at whatever state it was when I created it. In production, it would get refreshed weekly. My test conditions don’t match production conditions in the most important dimension — how stale the stable lock is relative to the real lock.

The cache zoo

Part of why I’m stuck is that I don’t fully understand all the caches in play. Let me try to enumerate them, as much for my own sanity as for yours:

1. Docker BuildKit layer cache — local to the runner’s Docker daemon. If a layer’s inputs haven’t changed, BuildKit reuses it. This is the cache we’re trying to optimize.

2. Registry-based cache (cache-to=type=registry,mode=max) — BuildKit pushes cached layers to our Azure Container Registry. When a different runner builds the same image, it can pull cached layers from ACR instead of rebuilding. This is what made the 60% speedup real in Part 2.

3. uv package cache (--mount=type=cache,target=/root/.cache/uv) — uv’s own download cache. Even if the layer rebuilds, uv can skip re-downloading packages it already has. But this is a BuildKit cache mount — it persists across builds on the same runner, but not across runners.

4. ccache for CUDA kernels — some of our dependencies compile CUDA kernels during install. ccache caches the compiled objects. We persist this through GitHub Actions cache.

5. GitHub Actions cache — stores the ccache artifacts between workflow runs. Has a 10GB limit per repo. Can be evicted.

6. Runner disk — ephemeral. The runner’s entire filesystem can be wiped between jobs, taking all local caches with it.

These caches interact in ways I haven’t fully mapped out. For example: if the registry cache has our Phase 1 layer, but the runner is cold (no local BuildKit cache), does BuildKit actually pull the Phase 1 layer from the registry and use it? I think so — that’s the whole point of cache-from=type=registry. But I haven’t verified it with my own eyes in our specific setup.

And then there’s the question of which cache is responsible for the 16-minute vs 28-minute spread on main. When main gets a 16-minute build, what hit? The local BuildKit cache? The registry cache? Both? If the 16-minute runs are local cache hits and the 28-minute runs are full registry pulls, then my optimization needs to beat 28 minutes (the cold-runner case), not 16 minutes (the warm-runner case).

I am drawing diagrams on napkins and none of them are helping.

The nondeterminism problem

Even if I understood all the caches perfectly, there’s a signal-to-noise problem.

Build times on main range from 16 to 28 minutes — a 12-minute spread. On my PR branch, I’ve seen 21 to 55 minutes (and the 55 is an outlier from runner contention, but how do I know the others aren’t also affected by contention I just can’t see?).

To statistically distinguish “27 minutes” from “28 minutes,” I’d need… a lot of runs. Dozens, probably. Each one takes 30 minutes and burns CI resources that the team needs for actual work. I’m not going to run 50 throwaway builds to get a p-value.

And even if I did, the number I’d be measuring is specific to this week’s dependency state, this week’s runner fleet, this week’s cache warmth. Next month it could be completely different.

This is the point where the scientist in me says “your experiment is underpowered” and the engineer in me says “just ship the thing.”

The “just ship it” argument

Here’s what I keep coming back to: what’s the worst case?

If the optimization doesn’t help — say uv-stable.lock gets stale, or the cache dynamics don’t work out, or whatever — then Phase 1 installs from the stable lock and Phase 2 reinstalls everything from the real lock. You run uv sync twice instead of once. That adds maybe 1-2 minutes to the build. Not zero cost, but not catastrophic.

The optimization degrades to baseline. It doesn’t make things worse in any meaningful way. The floor is “1-2 minutes slower than current,” and the ceiling is “10+ minutes faster.”

That’s actually a really nice property. Most optimizations have a failure mode where they make things worse — cache poisoning, wrong code paths, race conditions. This one just becomes a no-op. The extra uv sync is redundant but harmless. Like wearing a seatbelt on a day you don’t crash. Slight inconvenience, no downside.

The other comfort: it’s a Dockerfile change. If it causes problems, git revert and you’re back to the old build in one commit. The blast radius is one file. It doesn’t touch application code, test infrastructure, or deployment configs. The reversibility cost is approximately five seconds of typing.

What “ship to production” means

This is maybe the most honest thing I can say about this whole process: there is no A/B test infrastructure for Docker builds.

You can’t run the old and new Dockerfiles in parallel on production CI. You can’t gradually roll out a Dockerfile change to 10% of builds. You can’t canary it. You just… merge the PR. And then you watch the build times.

“Ship to production” means:

1. Merge PR #19942
2. Make sure the uv-stable.lock refresh workflow is set up and running
3. Watch the build dashboard for a week
4. Compare the distribution of build times to the previous week
5. Hope the comparison is meaningful despite all the confounders

That’s it. That’s the production validation plan. I’m a little embarrassed by how unsophisticated it is, but I also can’t think of anything better that’s worth the effort.

What I’m going to do

I’m going to merge it.

Not because I’ve proven it works in production. I haven’t, and I don’t think I can — not with the tools and time I have. I’m going to merge it because:

• The controlled experiment showed a clear signal (60% faster on rebuilds with push)
• The failure mode is benign (degrades to baseline + ~1-2 min)
• It’s trivially reversible
• The testing problem doesn’t get easier by waiting — it’s fundamentally about environmental state I can’t control in advance

After merging, I’ll do the boring part: watch build times for a week, compare them to the previous week, and see if the distribution shifted. If it did, great. If it didn’t, I’ll investigate whether the caches are behaving as expected. If it got worse somehow, revert.

The lesson, if there is one: some optimizations can’t be fully validated before shipping. You can validate the mechanism (we did, in Part 2). You can validate the safety (graceful degradation, easy revert). But you can’t validate the production effect without putting it in production. At some point, the most rigorous thing you can do is ship it and measure.

It’s unsatisfying. I like controlled experiments with clean results. But the alternative is leaving a likely-good optimization sitting in a PR indefinitely because I can’t achieve a level of certainty that the problem doesn’t allow for.

Sometimes “merge and watch” is the whole plan. And that’s fine.