I spent three days optimizing a Docker build so that dependency installs take 30 seconds instead of 20 minutes. The fix is about six lines of Dockerfile. The journey to get there involved staring at CI logs, learning things about uv I never wanted to know, and the humbling realization that I couldn’t actually benchmark the fix on my laptop. Let me save you the trip.
The problem: one line change, 20-minute penalty
We have a Python monorepo for training large language models. The Docker images install hundreds of packages — PyTorch, CUDA libraries, half of PyPI. The dependency install step takes 20-25 minutes on a cold run. That’s fine, because Docker layer caching means you only pay that cost when dependencies change.
Here’s the thing: dependencies change constantly.
Someone bumps numpy from 1.26.3 to 1.26.4. A transitive dependency updates. Someone adds a dev tool. Any of these changes touch uv.lock, our lockfile. And the moment uv.lock changes, Docker looks at the layer cache, says “nope, file changed,” and reinstalls everything from scratch. All 400+ packages. Even though 399 of them are identical.
It’s like burning down your house and rebuilding it because you wanted to change a lightbulb.
Docker layer caching, explained on a napkin
If you already know how Docker layer caching works, skip this. If you don’t, here’s the 30-second version.
A Dockerfile is a stack of instructions. Each instruction creates a “layer” — a snapshot of the filesystem at that point. Docker caches these layers. When you rebuild, it checks each layer top-to-bottom: “Has anything changed?” If a layer is identical to last time, Docker skips it and uses the cached version. Fast. Free.
But the moment one layer changes, every layer after it rebuilds too. The cache is a stack of Jenga blocks — pull one from the middle and everything above it falls.
Our Dockerfile looked like this:
COPY uv.lock /app/yolo/uv.lock
COPY pyproject.toml /app/yolo/pyproject.toml
RUN uv sync --package mai_job --frozen
Simple. Copy the lockfile, copy the project config, install deps. Three layers. The uv sync layer is cached as long as neither uv.lock nor pyproject.toml changes.
The problem is that uv.lock is a 10,000-line file that regenerates entirely when any dependency in the monorepo changes. Not just the ones this package uses — any dependency. Someone adds pytest-xdist to an unrelated package? uv.lock changes. Cache busted. 20 minutes.
The grocery store analogy
Imagine your weekly grocery run. You go to Costco every Sunday and buy 90% of what you need for the week — rice, chicken, vegetables, coffee, the bulk stuff that barely changes. That trip takes an hour, but you only do it once a week.
Then on Wednesday you realize you forgot cilantro. You don’t go back to Costco and redo the entire shopping trip. You pop into the corner store, grab the one thing, and you’re out in 5 minutes.
That’s the two-phase trick. Phase 1 is your Costco run. Phase 2 is the corner store.
The fix: two-phase dependency install
We split the dependency install into two steps:
# Phase 1: the Costco run (stable deps, cached for ~a week)
COPY pyproject-stable.toml /app/yolo/pyproject.toml
COPY uv-stable.lock /app/yolo/uv.lock
RUN uv sync --package mai_job --frozen
# Phase 2: the corner store (just the diff)
COPY pyproject.toml /app/yolo/pyproject.toml
COPY uv.lock /app/yolo/uv.lock
RUN uv sync --package mai_job --frozen --inexact
Here’s what’s happening:
uv-stable.lock is a second lockfile that we regenerate on a schedule — roughly weekly, or when someone explicitly bumps it. It contains the “stable” set of dependencies. Think of it as a snapshot of “what 90% of our deps looked like last Tuesday.” This file doesn’t change when someone bumps a minor version of some transitive dependency. It’s stable. That’s the whole point.
Phase 1 installs from this stable lockfile. Since uv-stable.lock rarely changes, this layer stays cached across most builds. The 20-minute install? Cached. Free. Skipped.
Phase 2 copies the real uv.lock and runs uv sync again, but with --inexact. This flag is the secret sauce.
Normally, uv sync looks at what’s installed and what the lockfile says, and if there’s any mismatch, it errors out or reinstalls everything. The --inexact flag says: “Hey, I know there might be extra packages installed that aren’t in my lockfile. That’s fine. Just install whatever’s missing and move on.” Since Phase 1 already installed 90% of the packages, Phase 2 only installs the diff. Maybe 3-5 packages. Maybe 30 seconds.
The three scenarios
Not every build benefits equally. There are three scenarios:
Scenario A: Nothing changed. Both lockfiles are the same as last build. Docker cache hits on Phase 1. Phase 2’s files also haven’t changed, so that’s cached too. Total dep install time: 0 seconds (fully cached). Same as before the optimization. No improvement, no regression.
Scenario B: Minor dep bump. Someone updated a dependency. uv.lock changed but uv-stable.lock didn’t. Phase 1: cached (stable lock unchanged). Phase 2: cache miss, but only installs the diff. This is where the magic happens. Instead of 20 minutes for a full reinstall, you get ~30 seconds for the delta. This is the scenario that happens 3-4 times a week in a monorepo.
Scenario C: Major dep overhaul / stable lock refresh. Both lockfiles changed. Phase 1 cache miss — full install. Phase 2 runs on top but there’s almost nothing to do since Phase 1 just installed fresh. Slightly slower than the old single-phase approach because of the overhead of running uv sync twice. Maybe 30-60 extra seconds. This happens roughly once a week.
The punchline: Scenario B is the one that matters, and it’s the one that happens most often. In a monorepo with dozens of contributors, someone is bumping something almost every day.
Why benchmarking this was harder than the fix
Here’s the part I didn’t expect. You can’t easily test this on your laptop.
The Docker builds target GPU runners. They pull from private Azure Container Registry. The build context needs Tailscale VPN. The base images have CUDA libraries. My MacBook has approximately zero of these things.
So the “benchmark” was: push the change, wait for CI to run, compare the timelines. Scientific method, meet “I’m going to squint at two GitHub Actions runs and call it data.”
I compared CI runs before and after the change for the Scenario B case (minor dep bump, which is the only scenario where the optimization matters). The dependency install step dropped from ~20 minutes to under a minute. The rest of the build (CUDA kernel compilation, etc.) was unchanged.
Could I have set up a local simulation with mock registries and fake base images? Sure. Would it have taken longer than just reading the CI logs? Absolutely.
The maintenance cost
There’s a catch — there’s always a catch. You now have two lockfiles to maintain.
uv-stable.lock needs periodic regeneration. Too infrequent and the delta between stable and real grows, making Phase 2 slower. Too frequent and you’re just busting the Phase 1 cache for no reason. Weekly feels right for our cadence.
We generate it with a scheduled CI job: run uv lock, save the output as uv-stable.lock, open a PR. Someone approves it, it merges, and the next Docker build picks up the fresh stable baseline. The pyproject-stable.toml follows the same lifecycle.
It’s not zero maintenance. But it’s “approve a bot PR once a week” maintenance, not “rearchitect your build system” maintenance.
The takeaway
The trick isn’t novel. It’s the same principle behind every caching optimization ever: separate the things that change from the things that don’t. Docker’s layer cache gives you a free caching layer, but only if your layers are structured so that frequently-changing content is as far down as possible.
When your lockfile is a single monolithic artifact that changes on every minor version bump, you’ve accidentally told Docker “rebuild everything, always.” The two-phase split is just a way of saying “actually, most of this is stable — here’s the proof.”
Six lines of Dockerfile. Three days of investigation. Thirty seconds instead of twenty minutes.
I’d do it again, but I’d bring snacks for the CI wait times.