Docker Builds: Four Tries, One Dockerfile

Nobody ships the perfect Docker optimization on the first try. I know this because I’ve now shipped four iterations of the same optimization on the same Dockerfile, and iteration three was a regression I didn’t catch for a week. Here’s the arc.

The starting point: death by lockfile

Our Python monorepo builds a pretraining Docker image with ~18,000 resolved packages. The dependency install step takes 20-28 minutes cold. Docker layer caching should protect us — install once, cache forever — except uv.lock changes on basically every PR. Someone bumps a dev tool, a transitive dep updates, whatever. Cache busted. Full reinstall. Every time.

The fix came in four rounds. Same person, same Dockerfile, progressively less wrong.

PR #21006: the two-phase split

The first PR attacked the fundamental problem: uv.lock is a monolithic file that changes constantly, but 95% of its contents are stable week-to-week.

The idea: maintain a second lockfile — uv-stable.lock — that snapshots uv.lock on a weekly cron. Install from that first (Phase 1), then run the real uv.lock on top (Phase 2).

# Phase 1: stable deps — cached for ~1 week
COPY uv-stable.lock /app/yolo/uv.lock
RUN uv sync --package mai_job --frozen

# Phase 2: just the diff
COPY uv.lock /app/yolo/uv.lock
RUN uv sync --package mai_job --frozen --inexact

The --inexact flag is the key. It tells uv: “don’t remove packages that aren’t in this lockfile — just install what’s missing.” Without it, Phase 2 would see 18,000 packages installed that aren’t in the real lockfile and start ripping them out.

A weekly GitHub Actions cron job copies uv.lock → uv-stable.lock and opens a PR. Someone approves it, it merges, and the Phase 1 cache resets with a fresh baseline.

Result: Cache-miss rebuilds dropped from ~12 minutes to ~90 seconds (with push). I built a whole A/B test framework to prove it. 86% faster on the production-representative config.

This was the big win. But it left something on the table.

The next bottleneck reveals itself

With uv package installs no longer the bottleneck, the next slowest thing became visible: CUDA kernel compilation.

Our monorepo has a package called mai_kernels — custom CUDA kernels that get compiled during uv sync --package mai_job. This compilation takes 5-10 minutes depending on ccache warmth. And here’s the problem: it happened in the same Docker layer as the final uv sync, which meant any lockfile change would bust that layer and trigger a full recompilation. Even if mai_kernels code hadn’t changed at all.

The two-phase trick fixed the uv install time. It didn’t fix the CUDA recompilation time. They were bundled in the same layer, so one was always dragging the other down.

The narrow mount experiment

The idea was straightforward: pull mai_kernels compilation into its own layer, before Phase 2. If mai_kernels code doesn’t change, that layer caches. If uv.lock changes, only Phase 2 reruns — and it skips the kernel compilation because the kernels are already built.

# Phase 1: stable deps (unchanged from PR #21006)
RUN uv sync --no-install-workspace --frozen --package mai_job

# NEW: compile mai_kernels early, in its own cached layer
RUN \
  --mount=type=cache,target=/root/.cache/ccache \
  --mount=type=bind,source=../../mai_kernels,target=/app/yolo/mai_kernels,rw \
  <<EOF
set -euo pipefail
uv sync --package mai_kernels --frozen
cp -a -r /root/.cache/ccache /root/.cache/ccache-cpy
EOF

# Phase 2: the diff (now with --inexact)
RUN uv sync --no-install-workspace --frozen --package mai_job --inexact

# Final: install workspace packages, skip redundant compilation
DISABLE_BUILD_EXT=1 uv sync --package mai_job --frozen

Two interesting choices here.

Narrow mount. Instead of mounting the whole repo, I mounted only source=../../mai_kernels. Theoretically better for caching — Docker only tracks changes to the mount source when deciding whether to invalidate the layer. Narrower mount = fewer files that could change = more cache hits.

--inexact on Phase 2. Same trick as Phase 1 — be additive, don’t remove what the previous layer installed.

DISABLE_BUILD_EXT=1. The final uv sync --package mai_job would normally try to compile mai_kernels again (because that’s what installing a package with a build extension does). This env var tells it “the extensions are already built, skip it.” Without this, you’d compile CUDA kernels twice for no reason.

Why the narrow mount was abandoned

The narrow mount was clever. And it was abandoned for the wrong reason.

uv sync --package mai_kernels doesn’t just need the mai_kernels/ directory. It needs the workspace pyproject.toml and the lockfile. uv operates in the context of a workspace — it reads the root pyproject.toml to understand the dependency graph, resolves from uv.lock, and installs.

Mounting only mai_kernels/ meant uv couldn’t see the workspace root. It errored out. The conclusion at the time: “narrow mounts don’t work, uv needs the whole repo.” The fix seemed obvious — mount everything.

That conclusion was wrong. But I wouldn’t figure that out for another week.

PR #22455 v1: mount everything

Same architecture as the experiment, but with the “lesson learned”:

# Phase 1: stable deps (unchanged)
RUN uv sync --no-install-workspace --frozen --package mai_job

# Early mai_kernels compilation — separate cached layer
RUN \
  --mount=type=cache,target=/root/.cache/ccache \
  --mount=type=bind,source=../../,target=/app/yolo,rw \
  <<EOF
set -euo pipefail
uv sync --package mai_kernels --frozen
cp -a -r /root/.cache/ccache /root/.cache/ccache-cpy
EOF

RUN mv /root/.cache/ccache-cpy /root/.cache/ccache

# Phase 2: real lockfile
RUN uv sync --no-install-workspace --frozen --package mai_job

# Final: workspace packages, skip redundant CUDA build
DISABLE_BUILD_EXT=1 uv sync --package mai_job --frozen

Three changes from the experiment:

Full repo mount instead of narrow mai_kernels/ mount. The thinking: uv needs workspace context, narrow mounts break, just give it everything.

Dropped --inexact from Phase 2. In the experiment, Phase 2 used --inexact to preserve what the mai_kernels layer installed. In practice, the mai_kernels layer installs packages into the environment and Phase 2’s uv sync --no-install-workspace installs the same packages. No conflict. --inexact wasn’t adding value — it was just adding cognitive load.

Kept ccache mount on the final step. The experiment removed ccache from the final uv sync since DISABLE_BUILD_EXT=1 should skip compilation entirely. The refined version keeps it as a safety net. Belt and suspenders.

This shipped. It looked clean. It was a regression.

“Wait, it’s slower?”

A week after shipping, I ran A/B tests. Same framework I’d used to validate PR #21006. Two scenarios: warm ccache (the common case) and cold ccache (worst case).

The optimization that was supposed to cache CUDA compilation was making builds slower. In both scenarios. Consistently.

The cold ccache number was the smoking gun. If the Docker layer were actually caching the mai_kernels compilation, ccache warmth shouldn’t matter — you’d skip the entire compilation step. The fact that cold ccache was 26.5% slower meant Docker was recompiling mai_kernels on every single build, and the cold ccache had nothing to fall back on.

The “cached” layer wasn’t caching. At all.

The root cause: BuildKit bind mount cache keys

Here’s the thing about --mount=type=bind that I didn’t fully appreciate.

When BuildKit evaluates whether a RUN instruction’s layer is cached, it looks at the instruction text and the content hash of the bind mount source. Not “did the files that the RUN command actually reads change.” The hash of the entire mount source directory.

--mount=type=bind,source=../../,target=/app/yolo,rw

The source here is ../../ — the entire repo root. And uv.lock lives at the repo root. So any lockfile change — the exact thing that happens on basically every PR — changes the content hash of the mount source, which changes the cache key, which invalidates the layer.

The mai_kernels compilation layer wasn’t cached because its bind mount included uv.lock. The whole point of the two-phase split was to isolate uv.lock churn from expensive operations. And then I mounted it right back into the expensive operation.

That’s why the narrow mount experiment (source=../../mai_kernels) was the right idea. A mount scoped to just mai_kernels/ doesn’t include uv.lock. Lockfile changes don’t bust the cache key. The CUDA compilation layer actually caches.

Why the narrow mount actually works

Back to the experiment that was abandoned. The narrow mount errored out because uv sync --package mai_kernels needs workspace context — pyproject.toml and uv.lock.

But here’s what I missed: those files were already in the Docker image.

Phase 1 does COPY uv-stable.lock /app/yolo/uv.lock followed by uv sync. The workspace pyproject.toml gets COPYed in even earlier. By the time we reach the mai_kernels layer, /app/yolo/pyproject.toml and /app/yolo/uv.lock already exist in the image filesystem.

The bind mount only needs to provide the source code. The workspace root, the lockfile, the dependency graph — all of that is already sitting there from earlier layers. You don’t need to bind-mount them. They’re already there.

The experiment failed because Phase 1 was cleaning up after itself — rm uv.lock removed the stable lockfile (remember, uv-stable.lock gets COPYed into the image as uv.lock) after Phase 1 finished, to keep the image tidy. So by the time the mai_kernels layer ran, the lockfile was gone. Not because narrow mounts are fundamentally broken. Because we deleted the file it needed.

PR #22455 v2: narrow mounts return

The fix had two parts.

Part 1: stop deleting uv-stable.lock. Phase 1 copies uv-stable.lock into the image as uv.lock. Previously, it ran rm uv.lock after the sync to keep things tidy. But the mai_kernels compilation step needs that lockfile. Remove the rm, and the stable lockfile stays in the image — available for the mai_kernels step. Phase 2 overwrites it with the real uv.lock anyway.

Part 2: narrow bind mounts. Mount only the source directories in mai_kernels’ workspace dependency chain — not the whole repo.

# Phase 1: stable deps (unchanged — but no longer deletes uv.lock after)
RUN uv sync --no-install-workspace --frozen --package mai_job

# Early mai_kernels compilation — narrow mounts, actually cached this time
RUN \
  --mount=type=cache,target=/root/.cache/ccache \
  --mount=type=bind,source=../../mai_kernels/,target=/app/yolo/mai_kernels,rw \
  --mount=type=bind,source=../../lib/mai_nano/,target=/app/yolo/lib/mai_nano,rw \
  --mount=type=bind,source=../../lib/blobfile/,target=/app/yolo/lib/blobfile,rw \
  <<EOF
set -euo pipefail
uv sync --package mai_kernels --frozen
cp -a -r /root/.cache/ccache /root/.cache/ccache-cpy
EOF

RUN mv /root/.cache/ccache-cpy /root/.cache/ccache

# Phase 2: real lockfile
RUN uv sync --no-install-workspace --frozen --package mai_job

# Final: workspace packages, skip redundant CUDA build
DISABLE_BUILD_EXT=1 uv sync --package mai_job --frozen

Three narrow mounts: mai_kernels/, lib/mai_nano/, lib/blobfile/. That’s the full workspace dependency chain for mai_kernels. Nothing else. No uv.lock in the mount source. No repo root. Just the source code that actually matters for compilation.

If uv.lock changes but mai_kernels code doesn’t? Cache hit. Docker skips the entire compilation. Phase 2 handles the lockfile diff. The layers are finally independent.

The results

Rebuilds dropped from ~2m45s to 66 seconds with zero variance. That’s the tell — zero variance means the CUDA compilation is fully cached. When it’s actually rebuilding, you get noise. When it’s a cache hit, you get the exact same time every run.

The evolution of --inexact

This flag has a fun arc across the iterations:

PR #21006: Essential. The whole two-phase trick depends on --inexact. Phase 2 needs to be additive — install what’s missing without removing what Phase 1 put there. Without this flag, there is no optimization.

Narrow mount experiment: Extended. Applied --inexact to Phase 2 again, plus used it to layer the mai_kernels packages on top. The flag was load-bearing in two places.

PR #22455 (both versions): Retracted (partially). Kept --inexact for the Phase 1→Phase 2 boundary where it’s essential, dropped it from the mai_kernels step where it wasn’t needed.

The pattern: when you’re exploring, you reach for the tool that works. When you’re refining, you figure out where it’s actually necessary.

The DISABLE_BUILD_EXT trick

This one’s worth calling out because it’s the kind of thing you only learn by watching a build fail in a confusing way.

uv sync --package mai_job installs mai_kernels as a workspace dependency. When uv installs a package with a build extension (C, CUDA, whatever), it runs the build extension. That’s the right behavior — you want your CUDA kernels compiled when you install the package.

But we already compiled them in the dedicated mai_kernels layer. Running the build extension again would:

1. Take 5-10 minutes for no reason
2. Potentially overwrite the cached artifacts with identical artifacts
3. Defeat the entire purpose of the dedicated caching layer

DISABLE_BUILD_EXT=1 is our project’s convention for “trust that the build extensions were already handled.” It’s not a uv flag — it’s an env var that our setup.py / build hooks check. If set, they skip compilation and assume the artifacts exist.

This is the glue between “compile mai_kernels in a cached layer” and “don’t redo that work when you install the full workspace.” Without it, you’d have two layers both trying to compile the same CUDA kernels.

What the four iterations tell you

There are two lessons here, and they pull in opposite directions.

The technical lesson: when you use --mount=type=bind, the content hash of the mount source becomes part of the layer cache key. Mount the whole repo for convenience, and you pay with cache stability. Every unrelated file change busts your “cached” layer. The fix is to mount the minimum — just the source directories that the operation actually needs.

The meta-lesson: when you abandon an approach because it “didn’t work,” make sure you understand why it didn’t work. The narrow mount experiment was abandoned because uv sync failed without workspace context. But the workspace context was already in the image — we’d just deleted it. The approach was right. The conclusion was wrong. And it cost a week of running a regression in production before I figured that out.

Four tries. Same Dockerfile. The first one was a breakthrough, the second was an experiment, the third was a mistake, and the fourth was the same experiment with one line removed.