Docker Builds: Mount Tricks and the uv-stable.lock Pattern

I spent a week optimizing Docker image builds for a Python monorepo that trains large language models. The Dockerfiles use BuildKit mounts extensively — --mount=type=cache, --mount=type=bind — and I realized I didn’t actually understand what either of them did. I’d been cargo-culting the syntax for months.

This post is everything I learned. It’s not a Docker tutorial — it’s the set of mental models I wish I’d had before touching any of this. If you’ve ever copy-pasted a --mount=type=cache line without knowing where the cache lives or why writes to it disappear, this is for you.

The problem: uv.lock changes too often

Our monorepo uses uv for Python package management. The uv.lock file pins every dependency — direct and transitive — and gets COPY’d into the Docker image early so we can install deps in a cached layer:

COPY uv.lock pyproject.toml /app/
RUN uv sync --no-install-workspace --frozen --package mai_job

The trick relies on Docker’s layer cache: if uv.lock hasn’t changed, Docker skips the RUN entirely and reuses the cached layer. Fast.

The problem: uv.lock had been modified 289 times in three months. Every time someone adds a dependency, tweaks a version pin, or even runs uv lock to resolve a conflict, the lockfile changes. And because it’s COPY’d into an early layer, every change invalidates the dep-install layer and everything after it.

On a cold build, uv sync downloads ~600 packages including PyTorch. That’s 5-10 minutes. On ARC runners (ephemeral Kubernetes pods), there’s no persistent Docker cache. Every build is cold. Every uv.lock change costs 5-10 minutes.

The solution: two-phase install with uv-stable.lock

The fix is a weekly snapshot. A scheduled CI job copies uv.lock to uv-stable.lock once a week. Then the Dockerfile does two install phases:

# Phase 1: Install from the stable snapshot (changes weekly)
COPY uv-stable.lock /app/uv.lock
COPY pyproject.toml /app/
RUN uv sync --no-install-workspace --frozen --package mai_job

# Phase 2: Catch up to the real lockfile (adds only what's missing)
COPY uv.lock /app/uv.lock
RUN uv sync --no-install-workspace --frozen --inexact --package mai_job

Phase 1 installs from uv-stable.lock. It changes once a week, so this layer stays cached ~95% of the time. Phase 2 copies the real uv.lock and runs uv sync --inexact. The --inexact flag tells uv: “don’t remove packages that aren’t in this lockfile, just add what’s missing.” So if 3 new packages were added since the weekly snapshot, Phase 2 installs only those 3.

The expected value: instead of re-downloading 600 packages on every uv.lock change, you download 3-5 packages. On a warm build, Phase 1 is instant (cached layer) and Phase 2 takes seconds.

The minimal diff to the Dockerfile is almost embarrassingly small. You’re literally adding one COPY line and one RUN line, and changing source=uv.lock to source=uv-stable.lock in the Phase 1 block.

The part where I got confused

The optimization itself is straightforward. What broke my brain was understanding how the existing Dockerfile worked — specifically, how --mount=type=cache and --mount=type=bind interact with Docker’s image layers. I kept asking: “wait, if I write a file inside a mounted path during a RUN step, does it end up in the image?”

The answer is no. But understanding why requires a mental model I didn’t have.

BuildKit mounts: bind vs. cache

BuildKit gives you two mount types that work completely differently. Both of them confused me for different reasons.

–mount=type=bind

RUN --mount=type=bind,source=uv.lock,target=/app/uv.lock \
    uv sync --frozen --package mai_job

A bind mount maps a path from the build context into the container during a single RUN step. You specify both source (where in your build context) and target (where in the container).

Key properties:

Temporary — exists only for the duration of that RUN instruction
Not baked into the image layer — content at the target path is not part of the layer output
Read-only by default — add ,rw to make it writable
Source is your build context — the files on disk that Docker was given (usually . in docker build .)

–mount=type=cache

RUN --mount=type=cache,target=/root/.cache/uv \
    uv sync --frozen --package mai_job

A cache mount is Docker managing a persistent volume for you. You only specify target — there is no source. Docker decides where it lives on the host. You never see the host path.

Key properties:

Persistent across builds — on the same Docker daemon / machine
Not baked into the image layer — same as bind, content at the target path is not part of the layer output
Docker manages lifecycle — it might get garbage-collected under disk pressure
No source parameter — Docker owns this volume entirely

The critical difference

	`type=bind`	`type=cache`
Source	Your build context (you specify path)	Docker-managed volume (no source)
Lifetime	This RUN step only	Persists across builds
In image?	No	No
Use case	Inject files without COPY (avoids layer bloat)	Persist download/compilation caches between builds

The shadow model

This is the mental model that made everything click. Both mount types shadow the image layer at the target path. Like putting a book on top of another book — the bottom one still exists, you just can’t see it.

During a RUN step with a mount:

Docker has the current image layer state (everything from previous layers)
The mount is placed on top of whatever exists at target, hiding it
All reads and writes to target go to the mounted volume, not the image layer
When the RUN step ends, the mount is removed
The original image layer at that path is visible again — untouched

Think of it like Python variable shadowing:

x = "image layer"       # The path exists in the image

def run_step():
    x = "cache volume"  # Mount shadows the image layer path
    print(x)            # → "cache volume"
    x = "new data"      # Writes go to the mount, not the image

run_step()
print(x)                # → "image layer" (unchanged!)

Or the desk analogy:

Your desk (image layer):     /root/.cache/uv/  → empty folder
Docker puts a box on top:    /root/.cache/uv/  → Docker's cache volume
                             You can't see the folder underneath.
                             All your work goes into the box.

RUN ends, Docker takes box away:
                             /root/.cache/uv/  → empty folder again
                             (The box still exists in Docker's storage,
                              ready for the next build.)

This is why --mount=type=cache works for build caches. uv downloads wheels into /root/.cache/uv during the RUN step. Those downloads go into Docker’s cache volume. The next build mounts the same volume — warm cache. But the downloads are never in the image itself.

The ccache smuggling trick

Once the shadow model clicked, I could finally understand the most confusing pattern in our Dockerfile:

RUN \
  --mount=type=cache,target=/root/.cache/ccache \
  --mount=type=bind,source=../../,target=/app/yolo,rw \
  <<EOF
uv sync --package mai_job --frozen
cp -a -r /root/.cache/ccache /root/.cache/ccache-cpy
EOF

RUN mv /root/.cache/ccache-cpy /root/.cache/ccache

What’s happening here, step by step:

uv sync --package mai_job compiles CUDA kernels. The compiler writes its cache to /root/.cache/ccache. But that path is shadow-mounted — writes go to Docker’s cache volume, not the image.
cp -a -r /root/.cache/ccache /root/.cache/ccache-cpy copies the ccache data from the mounted path to a non-mounted path. /root/.cache/ccache-cpy is not under any mount — it’s a regular image layer path. Writes here ARE baked into the image.
The RUN step ends. The cache mount at /root/.cache/ccache disappears. But /root/.cache/ccache-cpy survives — it’s in the image layer.
The next RUN mv renames it back to /root/.cache/ccache — now at the real image layer path (no mount shadowing this time).

This is smuggling. You’re pulling data out of a mounted volume into the image by writing to a path that isn’t under any mount.

Why bother?

Our CI runs on ARC runners — ephemeral Kubernetes pods. When the pod dies, Docker’s cache volumes die with it. The cache mount gives you speed when you rebuild on the same machine, but on ARC, every build is on a fresh pod. There IS no “same machine.”

The smuggling trick embeds the ccache directly in the image. Next build, even on a different pod, the base image already contains warm ccache data. Belt and suspenders — cache mount for when you get lucky with pod reuse, image-embedded cache for when you don’t.

Three experiments to prove it

If you’re still skeptical, run these yourself:

# Experiment 1: Write to mounted path → GONE
RUN --mount=type=cache,target=/tmp/stuff \
    echo "hello" > /tmp/stuff/file.txt
RUN cat /tmp/stuff/file.txt
# ❌ File not found — mount is gone, path is empty again

# Experiment 2: Write to non-mounted path → EXISTS
RUN --mount=type=cache,target=/tmp/stuff \
    echo "hello" > /tmp/other/file.txt
RUN cat /tmp/other/file.txt
# ✅ "hello" — written to image layer, not to a mount

# Experiment 3: The smuggling trick
RUN --mount=type=cache,target=/tmp/stuff \
    echo "hello" > /tmp/stuff/file.txt && \
    cp /tmp/stuff/file.txt /tmp/saved.txt
RUN cat /tmp/stuff/file.txt   # ❌ gone
RUN cat /tmp/saved.txt         # ✅ "hello" — smuggled out

Experiment 1 fails because the data is in the cache volume, not the image. Experiment 3 succeeds because cp moves data from the shadowed mount to a regular image path before the RUN ends.

COPY vs. bind mount: when does Docker invalidate cache?

This is a subtlety that matters for the two-phase optimization.

COPY creates a layer cache key:

COPY uv.lock /app/uv.lock
RUN uv sync --frozen

Docker hashes the contents of uv.lock when processing the COPY instruction. If the file changed since the last build, this layer and all subsequent layers are invalidated. That’s the whole mechanism that makes Docker layer caching work.

Bind mounts do not:

RUN --mount=type=bind,source=uv.lock,target=/app/uv.lock \
    uv sync --frozen

The bind mount is not part of the layer’s cache key. Docker doesn’t hash the bound file — it only checks whether the RUN instruction text changed. So even if uv.lock has completely different contents, Docker thinks the layer is still valid and serves the cached version.

This is why the two-phase optimization uses COPY for both lockfiles:

COPY uv-stable.lock /app/uv.lock    # Docker hashes this → cache key
RUN uv sync --frozen ...             # Cached if uv-stable.lock unchanged

COPY uv.lock /app/uv.lock           # Docker hashes this → new cache key
RUN uv sync --inexact --frozen ...   # Cached if uv.lock unchanged

If uv-stable.lock were bind-mounted instead of COPY’d, Docker would never know it changed, and you’d get stale layers silently. The COPY is doing double duty — putting the file in the image AND acting as a cache-invalidation trigger.

–no-install-workspace: the layer separation trick

The --no-install-workspace flag does exactly what it says — install everything except the workspace packages (your actual source code). Only third-party dependencies.

Why this matters for Docker: third-party deps change slowly (lockfile changes), but source code changes on every commit. If you install deps and source code in the same layer, a one-line code change re-downloads PyTorch. Splitting them into separate layers means:

# Layer 1: Third-party deps only (cached unless lockfile changes)
RUN uv sync --no-install-workspace --frozen --package mai_job

# Layer 2: Workspace packages + CUDA compilation (rebuilds on code change)
COPY . /app/
RUN uv sync --package mai_job --frozen

Layer 1 stays cached through code changes. Layer 2 rebuilds, but it only needs to compile your CUDA kernels — not re-download 600 packages.

Without --no-install-workspace, uv would try to install the workspace packages too, which means it needs the source code, which means you’d have to COPY source code before this layer, which means code changes invalidate this layer. The flag is what makes the separation possible.

uv cache prune –ci

You’ll sometimes see this after a uv sync:

COPY uv.lock pyproject.toml /app/
RUN uv sync --frozen --package mai_job && \
    uv cache prune --ci

When uv sync runs, it downloads wheels into /root/.cache/uv before installing them. If you used COPY for the lockfile (no cache mount on /root/.cache/uv), those downloaded wheels are baked into the image layer. They’re dead weight — the packages are already installed, the wheel files just sit there taking up space.

uv cache prune --ci cleans out this download cache. The --ci flag is aggressive — it removes everything that isn’t currently needed.

When does this NOT matter? When you use --mount=type=cache,target=/root/.cache/uv. The download cache is in the mount, which (as we now know) is never baked into the image. Nothing to prune.

Docker cache persistence on different runner types

All of the above assumes Docker’s cache volumes persist between builds. Whether that’s true depends entirely on your CI infrastructure:

Runner setup	Cache survives between builds?	Why
Bare metal (your own server)	Always	Same Docker daemon, same disk
Runner pool (5 machines)	1-in-5 chance per build	Random scheduling across machines
ARC / Kubernetes pods	Never	Pod dies after job → disk gone

This is why the ccache smuggling trick exists. On ARC runners, --mount=type=cache is essentially a no-op across builds — there is no “across builds” because each build gets a fresh pod with a fresh Docker daemon. The cache mount still works within a single build (if multiple RUN steps use the same cache), but it provides zero benefit between builds.

The smuggling trick compensates by putting cache data in the image itself. Every subsequent build starts from a base image that already has warm ccache, regardless of which pod it lands on.

For teams on persistent runners, the smuggling trick is unnecessary overhead. For teams on ephemeral infrastructure, it’s the difference between 10-minute and 2-minute CUDA compilations.

Summary of mental models

If you take away three things:

Mounts shadow image layers. Anything at the mount’s target path is hidden during the RUN step. Writes go to the mount, not the image. When the RUN ends, the mount disappears and the original image layer is visible again.
COPY creates cache keys; bind mounts don’t. Docker hashes COPY’d files to decide whether to invalidate layers. Bind-mounted files are invisible to the cache system. Use COPY when you want Docker to detect changes.
Smuggling = copying from a mounted path to a non-mounted path before the RUN ends. This is how you get data out of a cache mount into the image layer. It’s the workaround for ephemeral CI infrastructure where Docker’s cache volumes don’t persist.

The uv-stable.lock optimization is three lines of Dockerfile changes. Understanding why those three lines work required understanding all of the above. I don’t think that’s a problem with Docker — these are genuinely useful abstractions. They just aren’t explained well anywhere because most Docker tutorials stop at COPY and RUN.