Docker Builds: Splitting uv.lock for Faster Builds

Every time someone on our team adds a Python dependency, our Docker image build grinds through a full reinstall of hundreds of packages. We set out to fix that. What we found was surprising — the biggest win wasn’t where we expected.

The Setup

We work in a Python monorepo for training multimodal LLMs. We use uv for package management, and our pretraining Docker image has a pretty standard pattern:

COPY ../../uv.lock /app/yolo/uv.lock
COPY ../../pyproject.toml /app/yolo/pyproject.toml

RUN <<EOF
  set -euo pipefail
  uv venv --python 3.12 --system-site-packages ${VIRTUAL_ENV}
  uv sync --no-install-workspace --frozen --package mai_job
  rm uv.lock pyproject.toml
  uv cache prune --ci
EOF

The problem: uv.lock changes frequently. Every dependency add or update touches it. And because Docker layer caching works by invalidating everything downstream of a changed COPY, the entire uv sync step — installing every single package from scratch — reruns on every change. On a big ML project with heavy dependencies (think PyTorch, transformers, various CUDA libraries), that’s not fast.

The Idea: A Stable Lockfile

What if we split the install into two phases?

1. Phase 1 installs from a “stable” lockfile (uv-stable.lock) that only changes weekly. This layer stays cached almost all the time.
2. Phase 2 installs from the real uv.lock, but since most packages are already installed from Phase 1, it only needs to handle the diff.

# Phase 1: Install from stable snapshot (cached ~weekly)
COPY ../../uv-stable.lock /app/yolo/uv.lock
RUN <<EOF
  set -euo pipefail
  uv venv --python 3.12 --system-site-packages ${VIRTUAL_ENV}
  uv sync --no-install-workspace --frozen --package mai_job
  uv cache prune --ci
EOF

# Phase 2: Sync the real lockfile (only the diff)
COPY ../../uv.lock /app/yolo/uv.lock
RUN <<EOF
  set -euo pipefail
  uv sync --no-install-workspace --frozen --package mai_job
  rm uv.lock pyproject.toml
  uv cache prune --ci
EOF

A weekly GitHub Actions cron job copies uv.lock to uv-stable.lock and opens a PR. If nothing changed that week, no PR. Simple.

The concept is straightforward. Testing it properly? That’s where it got interesting.

Attempt 1: Cold Build Comparison

We set up a CI workflow that builds both variants (baseline from main, optimized from our branch) on identical 32-core runners.

First test: just build both from scratch once.

Result: the optimized version was slower. ~949 seconds vs ~823 seconds.

This makes sense if you think about it for two seconds, which we did not do before running the experiment. On a cold build with no cache, the optimized Dockerfile runs uv sync twice — once for the stable lock, once for the real lock. Of course it’s slower. The optimization only helps on rebuilds where the Phase 1 layer is cached.

Back to the drawing board.

Attempt 2: Warm Cache + Rebuild

Better experimental design this time. Each variant runs two passes:

1. Pass 0 (warm): Build from scratch, populating the Docker buildx cache.
2. Simulate a lockfile change: Append # cache-bust to uv.lock.
3. Pass 1 (rebuild): Build again with the modified lockfile.

This simulates the real scenario — you have a cached build, someone changes a dependency, and you rebuild.

Results without --push:

No improvement. Actually slightly worse.

We stared at this for a while.

Then we added --push (to actually push the image to our container registry, like CI does in the real world):

~60% faster. Now we’re talking.

The Insight: It’s the Upload, Not the Install

This was the key finding, and it wasn’t obvious going in.

The compute time for uv sync on a rebuild is relatively small — most packages are already on the runner’s cached filesystem. The expensive part is pushing the resulting layers to the container registry.

In the baseline, when uv.lock changes, the COPY layer invalidates, and the entire RUN layer after it — containing every installed package, potentially gigabytes of CUDA libraries and ML frameworks — has to be re-uploaded to the registry. Even if 99% of the packages are identical to what’s already there.

In the optimized version, the big Phase 1 layer (stable lockfile) is already in the registry. Only the small Phase 2 layer (the diff) needs uploading. The total bytes going over the wire drops dramatically.

This also explains why we saw no improvement without --push. Locally, there’s nothing to upload — you’re just writing layers to disk, which is fast regardless. The optimization is specifically about network transfer to the registry.

A Subtle Detail: Cache Pruning

One thing that tripped us up briefly: you need uv cache prune --ci at the end of each phase, not just at the end.

Without it, Phase 1 downloads packages into the uv cache, and that cache gets baked into the layer. Phase 2 then inherits a bloated cache layer on top of the bloated Phase 1 cache. The --ci flag is more aggressive than plain uv cache prune — it removes everything not needed for the current venv state.

In a single-phase Dockerfile this is just good hygiene. In a two-phase setup, it’s critical for keeping layer sizes reasonable.

Keeping the Stable Lock Fresh

The refresh mechanism is deliberately boring:

on:
  schedule:
    - cron: "0 0 * * 1"  # Every Monday at midnight UTC
  workflow_dispatch:       # Manual trigger if needed

The workflow does cp uv.lock uv-stable.lock, checks if anything actually changed, and opens a PR if so. If uv.lock hasn’t changed since last Monday, nothing happens.

One edge case to be aware of: if this workflow fails silently for a few weeks (maybe the bot token expires, or someone changes branch protections), uv-stable.lock gets stale, and Phase 2 gradually becomes as expensive as the original single-phase approach. It degrades gracefully though — you never get a worse result than baseline, just a less improved one.

What We Used for Caching

This optimization depends on your Docker cache setup. We use docker buildx bake with HCL target definitions, and our cache is configured with mode=max. That’s important — mode=max stores every intermediate layer in the registry cache, not just the final image layers. Without it, the Phase 1 layer wouldn’t be cached in the registry, and the whole scheme falls apart.

If you’re using mode=min (the default), you’d only cache the final image’s layers, and intermediate build stages get thrown away. Check your cache-to configuration.

Open Questions

This experiment gave us a clear signal, but there are things we haven’t fully validated yet:

Image size. The two-phase approach adds an extra layer. In theory, the overlay filesystem means you’re paying for some duplicated metadata. In practice, with uv cache prune --ci after each phase, the overhead should be small — but we haven’t measured it carefully.

Real vs. simulated changes. We tested by appending a comment to uv.lock. Real dependency changes (adding torch 2.6, removing a deprecated package) might produce different cache behavior. The uv sync diff could be larger or smaller than our simulation suggests.

Other image targets. Our monorepo has multiple Docker images. Some use --mount=type=bind for the lockfile instead of COPY, which means the layer caching dynamics are fundamentally different. The two-phase trick doesn’t directly apply there — you’d need a different approach.

Should You Do This?

If you’re using uv in Docker and your builds push to a registry (so, basically any CI setup), and your lockfile changes more often than your stable dependency set, this is worth trying. The implementation cost is low — one extra COPY/RUN block in your Dockerfile and a simple cron workflow.

The mental model is: separate your frequently changing cache key (the real lockfile) from your rarely changing cache key (the stable snapshot). Let Docker’s layer caching do what it’s good at.

The surprising part, at least for us, was that the win comes primarily from the registry push, not from the package installation compute. If we’d only tested locally, we would have concluded the optimization doesn’t work and moved on. Sometimes the most important thing about an experiment is making sure you’re measuring the right thing.