Docker Builds: Layer Caching Trick for Monorepo Lockfiles

Every time someone on the team bumped a dependency, our Docker image rebuild took 15 minutes. Most of that time was reinstalling packages that hadn’t actually changed. Here’s the trick we used to bring it down to 3 minutes — and how we proved it worked.

Some Vocabulary First

If you’re new to this world, here’s a quick glossary. Skip ahead if you already know these.

• Docker image — A snapshot of an entire operating system + your code + all its dependencies, packaged up so it runs identically on any machine. Think of it like a .zip file of a fully configured computer.
• Dockerfile — The recipe that tells Docker how to build an image. Each line is an instruction: “copy this file,” “run this command,” etc.
• Layer — Each Dockerfile instruction creates a layer. Docker stacks them like pancakes. Layers are cached, so unchanged instructions are skipped on rebuild.
• Monorepo — One big repository containing many projects/packages, shared by many teams. Ours has dozens of Python packages.
• Lock file (uv.lock) — A machine-generated file that pins the exact version of every dependency your project uses, including transitive ones (dependencies of your dependencies). You don’t edit it by hand — you commit it and let the tooling manage it. It exists so that uv sync produces the same environment on every machine, every time.
• uv — A fast Python package manager. uv sync reads the lock file and installs exactly what it says. --frozen means “don’t re-resolve anything, just install.”
• BuildKit — The modern build engine behind docker build. It replaced the old builder and added features like cache mounts and bind mounts (explained later).
• CI (Continuous Integration) — Automated systems that build and test your code on every push. In our case, GitHub Actions.
• ccache — A compiler cache. It remembers previously compiled files so recompilation is near-instant when source code hasn’t changed.
• CUDA — NVIDIA’s programming model for GPU computation. Compiling CUDA code is slow (10-30 minutes from scratch).

The Setup: A Monorepo With a Giant Lock File

We work in a Python monorepo with dozens of packages. We use uv as our package manager, and it generates a single uv.lock file that pins every dependency across the entire repo. Ours is about 18,000 lines long.

The problem is that in a monorepo, lots of people touch lots of packages. Someone adds a dev dependency to an unrelated package? uv.lock changes. Someone bumps pytest? uv.lock changes. These changes are usually tiny — a few lines in an 18,000-line file — but they’re changes.

And Docker really cares about changes.

Why Docker Makes This Painful

Docker builds images in layers. Each instruction in a Dockerfile — COPY, RUN, etc. — produces a layer. Docker caches these layers and only reruns an instruction if its inputs changed.

Here’s the catch: if one layer’s inputs change, every layer after it is invalidated too. The cache is sequential — like a chain of dominoes. Knock one over, and everything downstream falls.

Our old Dockerfile looked roughly like this:

COPY uv.lock /app/yolo/uv.lock
COPY pyproject.toml /app/yolo/pyproject.toml
RUN uv sync --frozen --package mai_job

Here’s what happens when any line in uv.lock changes:

1. The COPY uv.lock layer sees different content → cache miss
2. The RUN uv sync layer is after it → also invalidated (domino effect)
3. uv sync reinstalls everything from scratch → 12-15 minutes

Someone on a totally different team adds a test utility to their package. Your pretraining image rebuilds for 15 minutes. That’s the pain.

The Trick: A Stable Snapshot

The insight is simple: most of the lock file doesn’t change week to week. The churn is at the edges — a few packages get bumped, a few new ones get added. The core 95% is stable.

So we split the install into two phases using two lock files:

• uv-stable.lock — A weekly snapshot of uv.lock. Updated every Monday by an automated cron job. Changes ~once a week.
• uv.lock — The real, always-current lock file. Changes constantly.

The new Dockerfile:

# Phase 1: Install from the stable snapshot (cached most of the time)
COPY uv-stable.lock /app/yolo/uv.lock
RUN uv sync --no-install-workspace --frozen --package mai_job

# Phase 2: Catch up to the real lock file (only installs the diff)
COPY uv.lock /app/yolo/uv.lock
RUN uv sync --no-install-workspace --frozen --package mai_job --inexact

Phase 1 installs 95% of your dependencies from the stable snapshot. Since uv-stable.lock only changes weekly, this layer stays cached almost all week. Docker sees the same file contents → cache hit → skip the whole thing in seconds.

Phase 2 copies in the real uv.lock and runs uv sync again — but with the --inexact flag.

(--no-install-workspace means “only install third-party dependencies, not our own monorepo packages.” We install those later.)

What --inexact Does

This is the key flag that makes the whole thing work.

Normally, uv sync is exact — it ensures your environment matches the lock file precisely. If something is installed that shouldn’t be, it removes it. If something is missing, it installs it. Full reconciliation.

--inexact changes the behavior to additive only. It installs anything that’s missing or needs updating, but it doesn’t remove extras. Think of it as “just fill in the gaps.”

Since Phase 1 already installed 95% of what you need, Phase 2 with --inexact only has to install the handful of packages that changed since the last weekly snapshot. Instead of reinstalling 500 packages, you’re installing 3.

And the final result is identical. The container always ends up reflecting the real uv.lock. The stable snapshot is purely a caching optimization — it never produces a wrong image. Worst case, if uv-stable.lock is really out of date, Phase 2 just installs a larger delta. Still correct, just slower.

The Weekly Auto-Update

uv-stable.lock gets refreshed by a GitHub Actions cron job:

name: Update uv-stable.lock
on:
  schedule:
    - cron: '0 0 * * 1'  # Every Monday at midnight UTC
  workflow_dispatch: {}    # Manual trigger if needed

jobs:
  update:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - run: cp uv.lock uv-stable.lock
      - # ... create PR if there are changes

That’s it. Copy the file, open a PR. No human intervention needed.

The cron syntax 0 0 * * 1 reads as: “minute 0, hour 0, any day of month, any month, day-of-week 1 (Monday).” workflow_dispatch adds a button in the GitHub UI to trigger it manually if you want to refresh mid-week.

How We Proved It Works

“It should be faster” isn’t good enough. We wrote a benchmark CI workflow that runs both approaches side-by-side on identical runners.

The methodology:

1. Warm build — Build the image once to populate Docker’s layer cache
2. Simulate a lock file change — Append a comment to uv.lock (mimics someone bumping a dependency)
3. Rebuild — Time how long the rebuild takes
4. Repeat 3 times — Because one data point is anecdote, not evidence

Both approaches run on the same runner spec (ubuntu-2404-x64-32c.128g.1200g — 32 cores, 128 GB RAM, 1.2 TB disk) to keep the comparison fair.

The results (for the two-phase lock file trick alone — no separate CUDA layer):

The warm build is roughly the same — both approaches do the same total work when nothing is cached. But on rebuilds after a lock file change, the optimized approach is ~4x faster. That’s ~10 minutes saved per rebuild.

Multiply that across every CI run, every developer building locally, every day — it adds up fast.

The Full Dockerfile

Here’s the actual working Dockerfile with the two-phase lock trick, plus the workspace install that handles CUDA compilation:

# Phase 1: Stable deps — cached layer (uv-stable.lock changes ~weekly)
COPY ../../uv-stable.lock /app/yolo/uv.lock
RUN <<EOF
  set -euo pipefail
  uv venv --python 3.12 --system-site-packages ${VIRTUAL_ENV}
  uv sync --no-install-workspace --frozen --package mai_job
  uv cache prune --ci
EOF

# Phase 2: Incremental sync with real lockfile (only installs diff)
COPY ../../uv.lock /app/yolo/uv.lock
RUN <<EOF
  set -euo pipefail
  uv sync --no-install-workspace --frozen --package mai_job --inexact
  rm uv.lock pyproject.toml
  uv cache prune --ci
EOF

# Install workspace packages (includes mai_kernels compilation via ccache)
RUN \
  --mount=type=cache,target=/root/.cache/ccache \
  --mount=type=bind,source=../../,target=/app/yolo,rw \
  <<EOF
set -euo pipefail
uv sync --package mai_job --frozen
cp -a -r /root/.cache/ccache /root/.cache/ccache-cpy
python -c "import flash_attn; print(flash_attn.__file__)"
EOF

RUN mv /root/.cache/ccache-cpy /root/.cache/ccache

A few things worth explaining if this syntax looks unfamiliar:

• <<EOF ... EOF — This is a heredoc. It lets you write multi-line shell scripts inside a single RUN instruction. Everything between <<EOF and EOF is one script. This keeps it as one Docker layer instead of multiple.
• set -euo pipefail — A shell safety net. -e = exit on any error. -u = error on undefined variables. -o pipefail = a piped command fails if any part fails. Without this, a failing uv sync might silently continue and produce a broken image.
• uv cache prune --ci — Removes downloaded package files that aren’t linked to the current environment. Without this, hundreds of MB of cached wheels would bloat the image. The --ci flag makes it extra aggressive.
• --mount=type=cache and --mount=type=bind — BuildKit features explained in detail in the cache field guide below.
• cp -a -r /root/.cache/ccache /root/.cache/ccache-cpy — The “ccache dance,” also explained below.

Phase 1 and Phase 2 are the lock file trick from the previous section. The final uv sync --package mai_job installs the actual workspace packages (the monorepo’s own code), which includes compiling CUDA extensions like mai_kernels and flash_attn. The python -c "import flash_attn" at the end is a smoke test — if the extensions weren’t properly compiled, this blows up during the build instead of at runtime.

The Optimization That Didn’t Work: CUDA as a Separate Layer

This is the part where I tell you about something we tried that made things worse. I think it’s more useful than only showing the version that worked.

We looked at that Dockerfile and thought: “CUDA kernel compilation is expensive. What if we pull it into its own layer between Phase 1 and Phase 2? Then uv.lock changes wouldn’t retrigger the CUDA build.”

The idea was to add a dedicated layer:

# Phase 1: Stable deps (same as above)
# ...

# NEW: Compile mai_kernels (CUDA) — separate layer so uv.lock changes don't retrigger
RUN \
  --mount=type=cache,target=/root/.cache/ccache \
  --mount=type=bind,source=../../,target=/app/yolo,rw \
  uv sync --package mai_kernels --frozen

# Phase 2: Incremental sync (same as above)
# ...

# Final workspace sync — skip extension rebuild since they're already compiled
RUN \
  --mount=type=bind,source=../../,target=/app/yolo,rw \
  DISABLE_BUILD_EXT=1 uv sync --package mai_job --frozen

Sounds reasonable. CUDA compilation isolated from lock file churn. Separate cache lifetime. Clean separation of concerns.

It made rebuilds slower. 8 minutes instead of 5. A net negative.

Why it didn’t work

The problem is that we were adding a Docker layer to cache something that was already cached by lower-level mechanisms:

1. uv’s own cache-keys. In mai_kernels/pyproject.toml, uv tracks source file changes via cache-key globs (**/*.cu, **/*.cpp, **/*.h, etc.). When you run uv sync --package mai_job and mai_kernels is a dependency, uv checks those globs. If the CUDA source files haven’t changed, uv skips the mai_kernels build entirely — regardless of whether uv.lock changed. The lock file tells uv which version to install, but the cache-keys tell it whether the build is stale.

2. ccache. Even when uv does trigger a rebuild, ccache makes recompilation near-instant by reusing previously compiled object files. CUDA compilation is slow from scratch (10-30 minutes), but with a warm ccache, it’s a few seconds.

So the Docker layer was solving a problem that didn’t exist. uv+ccache already meant that CUDA compilation was either skipped entirely or fast. The separate layer just added overhead: two uv sync calls instead of one, each with its own dependency resolution and virtual environment scanning. That overhead was about 3 minutes — more than the “optimization” could ever save.

The lesson

Before adding a Docker layer to cache something, check if your build tools already cache it. Docker layer caching is coarse-grained — it operates at the level of entire RUN steps. Build tool caching (uv’s cache-keys, ccache) is fine-grained — it operates at the level of individual source files. Adding a coarse-grained cache on top of a fine-grained one just adds overhead without providing any additional cache hits.

You can verify this yourself: do a warm build with the single uv sync Dockerfile, bust only uv.lock (append a comment), and rebuild. Check the logs — you’ll see uv skip the mai_kernels build entirely and go straight to installing the lockfile diff. The CUDA compilation never happens because uv knows the source files didn’t change.

A Field Guide to Docker’s Caching Layers

There are several different caching mechanisms at play in a Dockerfile like ours. Understanding how each one works — and what it doesn’t do — is useful whether you’re debugging a cache miss or (like us) trying to decide whether adding another caching layer is worth it.

1. Docker Layer Cache

This is the foundational one. Every Dockerfile instruction produces a layer, and Docker caches them. But the cache keys are computed differently depending on the instruction type:

• COPY/ADD: Cache key = content hash of the source files. Docker checksums every byte. One changed character → cache miss.
• RUN: Cache key = the literal command string + the hash of all parent layers. Docker does NOT look at what the command actually produces or what files it reads at runtime. Same command text, same parent layers → cache hit. Every time.

That second point surprises people. If your RUN step does apt-get update && apt-get install curl, Docker doesn’t know that the package repository changed overnight. It just sees the same command text and replays the cached result. This is why Docker caching is fast but sometimes gives you stale results.

And the sequential invalidation: the cache is a chain, not a tree. If layer 3 out of 10 invalidates, layers 4 through 10 all rebuild — even if their inputs are identical to the last build. Docker can’t skip ahead. This is exactly why a single COPY uv.lock → RUN uv sync is so expensive. Any lockfile change knocks over the first domino.

Where it lives: In the local BuildKit daemon’s storage. On CI runners, this is ephemeral — it’s gone when the runner is recycled after your job finishes.

2. Registry Cache (--cache-from / --cache-to)

The layer cache is great locally, but CI runners are ephemeral — each job gets a fresh machine with no build history. Every build would start completely cold. Registry cache solves this.

The idea: after building, push the layer cache to a container registry (like Docker Hub, or in our case Azure Container Registry). Before the next build, pull that cache back. Now your CI build starts warm.

Our CI does this:

docker buildx bake \
  --set "pretraining.cache-from=type=registry,ref=registry.example.com/cache:pr-123" \
  --set "pretraining.cache-from=type=registry,ref=registry.example.com/cache:main" \
  --set "pretraining.cache-to=type=registry,ref=registry.example.com/cache:pr-123,mode=max"

Two cache-from entries, tried in order. First your PR’s own cache (for rebuilds within the same PR), then main as a fallback. First build on a new PR? Falls back to main — warm start instead of cold.

mode=max is important. Without it, BuildKit only exports cache for layers in the final image. With mode=max, it exports all intermediate layers — including the uv sync steps. Without this, our whole optimization would fail on CI because the intermediate layer caches wouldn’t survive between runs.

Where it lives: In the container registry, stored as specially-tagged image manifests. Persists until overwritten.

3. BuildKit Cache Mounts (--mount=type=cache)

A cache mount is a persistent directory that BuildKit injects into a RUN step. It survives across builds, but — and this is the key part — it is NOT included in the final image.

RUN --mount=type=cache,target=/root/.cache/ccache \
    uv sync --package mai_job --frozen

The /root/.cache/ccache directory here persists in BuildKit’s internal storage. Next build, same contents are there — even if the RUN step itself needs to re-execute (cache miss). Think of it as a side-channel that lives outside the normal layer cache.

This is how we keep the compiler cache warm across builds. Even when the workspace install layer rebuilds, ccache inside the mount still has previously compiled object files.

The catch: Since cache mounts aren’t in the final image, anything you want to keep in the running container must be copied out. That’s the “ccache dance”:

# Inside the RUN step: copy ccache data from the mount to the layer filesystem
cp -a -r /root/.cache/ccache /root/.cache/ccache-cpy

# Separate RUN: move it to the real path (the mount is gone by now)
RUN mv /root/.cache/ccache-cpy /root/.cache/ccache

Why two steps? Inside the RUN step, /root/.cache/ccache is the mount — writing to it writes to the mount, not to the layer. So we copy to a different path (-cpy) that’s on the real filesystem. After the step ends, the mount disappears, and we rename the copy back.

On CI: Cache mounts live inside the BuildKit daemon, which dies when the CI runner is recycled. To bridge this gap, we use a GitHub Action called buildkit-cache-dance that syncs cache mount contents to/from GitHub Actions cache storage (a persistent key-value store that survives across workflow runs). It’s a workaround for the fact that these two cache systems have no native interop.

Where it lives: BuildKit daemon storage locally. GitHub Actions cache on CI (via buildkit-cache-dance).

4. BuildKit Bind Mounts (--mount=type=bind)

This is the counterintuitive one.

RUN --mount=type=bind,source=../../,target=/app/yolo,rw \
    uv sync --package mai_job --frozen

A bind mount makes files from the build context available inside the RUN step without COPY-ing them. You might expect: “If I change a file in the mounted directory, this RUN step should rebuild, right?”

Nope. Bind mounts do not affect cache keys. BuildKit computes the RUN step’s cache key from the command string and the parent layer hash — it does not hash the contents of bind-mounted directories. The mounted files could be completely different, and Docker would still cache-hit.

This is by design. Imagine hashing an entire monorepo on every RUN step — it’d be prohibitively expensive. The mental model is: COPY = “I depend on these files” (tracked, affects cache). Bind mount = “I need temporary access to these files” (untracked, invisible to cache).

For our workspace install step, this means the layer stays cached as long as the RUN command text and parent layers don’t change. When we do need to force a rebuild (e.g., after modifying CUDA source files), we pass --no-cache to the build command.

Also worth noting: the rw flag lets the command write to the mounted directory, but those writes are discarded after the step finishes. They go to a copy-on-write overlay, not the actual source files on disk. Your repo is safe.

Where it lives: Nowhere persistently — it’s a live view into the build context that exists only during the RUN step.

5. uv’s Package Cache and Cache-Keys

uv has its own internal cache of downloaded wheels and sdists. When uv sync downloads a package, it caches the download so future syncs don’t re-fetch. Standard package manager behavior — pip, npm, cargo all do this.

But uv also has a subtler and more powerful mechanism: cache-keys. In a package’s pyproject.toml, you can declare file globs that uv watches:

[tool.uv]
cache-keys = [{ file = "**/*.cu" }, { file = "**/*.cpp" }, { file = "**/*.h" }]

This tells uv: “This package needs to be built from source. Consider it stale only if files matching these globs have changed.” So uv sync will skip rebuilding mai_kernels unless the actual CUDA/C++ source files changed — even if uv.lock changed for completely unrelated reasons.

This is finer-grained caching than anything Docker can do at the layer level. Docker invalidates an entire RUN step. uv’s cache-keys check individual source files. This is why our CUDA-layer experiment failed — uv was already handling it.

The uv cache prune --ci calls after each sync step are about keeping the image small:

RUN <<EOF
  uv sync --no-install-workspace --frozen --package mai_job
  uv cache prune --ci
EOF

Without pruning, every downloaded wheel would persist in the Docker layer and ship with the image. --ci aggressively removes cached downloads that aren’t needed by the installed environment.

Where it lives: In the image layer’s filesystem, pruned to near-zero before the layer is committed.

6. ccache (Compiler Cache)

ccache sits in front of the C/C++/CUDA compiler. When your build system says “compile this file,” ccache intercepts the call, hashes the preprocessed source code (after macro expansion and #include resolution), the compiler flags, and the compiler version. If it’s seen that exact combination before, it returns the cached .o (object) file instantly instead of invoking the actual compiler.

This matters because nvcc (NVIDIA’s CUDA compiler) is slow. Compiling CUDA kernels from scratch can take 10-30 minutes. ccache makes recompilation near-instant on cache hit — seconds instead of minutes.

The setup is just environment variables:

ENV CCACHE_DIR=/root/.cache/ccache
ENV CMAKE_C_COMPILER_LAUNCHER=ccache
ENV CMAKE_CXX_COMPILER_LAUNCHER=ccache
ENV CMAKE_CUDA_COMPILER_LAUNCHER=ccache

These tell CMake (the build system that compiles our CUDA code): “Before invoking any C, C++, or CUDA compiler, pass the call through ccache first.” CMake still thinks it’s calling gcc or nvcc directly — ccache transparently wraps them.

On CI, the ccache data flows through three systems to stay alive:

1. GitHub Actions cache (actions/cache) — Stores ccache data persistently on GitHub’s servers across workflow runs, keyed by a hash of the CUDA source files
2. buildkit-cache-dance — Injects that data into BuildKit’s cache mount before the build starts
3. The in-Dockerfile copy trick — Bakes ccache data into the image layer so the running container has a warm compiler cache

Three levels of persistence for one compiler cache. Sounds over-engineered, but each level exists because the previous one has a gap: GitHub Actions cache doesn’t talk to BuildKit. BuildKit cache mounts don’t persist in the image. The image layer doesn’t persist across builds.

Where it lives: Simultaneously in GitHub Actions cache, BuildKit cache mount, and the final image layer.

How They All Stack Up

Here’s what happens on a typical PR build where uv.lock changed but CUDA sources didn’t:

Multiple caching systems, each covering a different gap. The key insight we took away: the fine-grained caches (uv cache-keys, ccache) are often more valuable than the coarse-grained ones (Docker layers). Know what your tools already cache before adding more layers.

Why Not Just Use Multi-Stage Builds?

Multi-stage builds solve a different problem. They let you have a “builder” stage and a slim “runtime” stage, reducing image size. But the bottleneck here isn’t image size — it’s the uv sync step re-executing when it doesn’t need to.

The issue is purely about Docker layer cache invalidation. Multi-stage builds don’t help with that because the lock file change would still invalidate the install layer in the builder stage. You’d have the same 15-minute rebuild, just in a different stage.

The Grocery Store Analogy

If all the Docker/layer/cache talk is still fuzzy, here’s the mental model:

You have a grocery list of 500 items. Every time anyone on your team changes even one item on the list, you throw away everything in your cart and re-shop the entire list from scratch. Takes an hour.

The optimization: keep a “baseline grocery list” that only updates weekly. Monday morning, you shop that list — 500 items, full trip, takes an hour. The rest of the week, you keep a second list of just the 2-3 items that changed since Monday. You only shop those.

Instead of re-shopping 500 items every day, you’re shopping 2-3. The final cart is identical either way. And if Monday’s list gets really stale? You just shop a slightly bigger delta list. Still correct, never wrong.

Takeaways

The pattern generalizes beyond uv and Python:

• Any lock file in a monorepo (npm’s package-lock.json, Cargo’s Cargo.lock, Go’s go.sum) has this problem. The lock file changes more often than your actual dependencies do.
• Stable snapshots + delta syncs work whenever your package manager supports additive installs. For uv, it’s --inexact. For npm, npm install is additive by default. Check your tool’s docs.
• Know what your tools already cache. Before adding a Docker layer to cache something, check if your build tools handle it at a finer granularity. Docker layer caching operates at the level of entire RUN steps. Build tool caching (uv’s cache-keys, ccache) operates at the level of individual source files. Adding a coarse-grained cache on top of a fine-grained one just adds overhead. We learned this the hard way with the CUDA layer experiment.
• Benchmark before and after. “It should be faster” isn’t a result. Run both approaches side-by-side, same hardware, multiple trials. Show the numbers. This applies double to “optimizations” — our CUDA layer looked smart on paper but added 3 minutes to every rebuild.
• Automate the snapshot refresh. If the stable snapshot requires manual updates, someone will forget and the delta will grow until it’s no better than the baseline. Cron job + auto-PR = zero maintenance.

The trick isn’t clever. It’s almost embarrassingly simple. But it shaved 10 minutes off every rebuild, and in CI, that’s the kind of thing that compounds into days of developer time saved per month. And the failed CUDA layer experiment? That saved us from shipping a slower Dockerfile to the team, which is its own kind of win.