Docker Builds: When One Cache Hides Another

I built a Docker layer optimization. I built an A/B test workflow to prove it works. The numbers came back suspiciously similar. For a minute I thought the optimization was broken. Then I realized: a different cache was doing the optimization’s job for it, and my test couldn’t tell the difference.

This is a story about caches hiding caches.

The setup

Quick context if you’re joining mid-series. We have a Python monorepo with Docker images that compile CUDA kernels (Flash Attention, custom ops — the usual ML training stack). The build has two expensive parts:

1. uv sync — installing Python dependencies (~20 min cold)
2. mai_kernels compilation — building C++/CUDA extensions (~15 min cold)

The first post covered how we split uv sync into a stable-lockfile phase and a diff phase, so that minor dependency bumps don’t trigger a full reinstall. The second post covered the A/B test workflow that proves the optimization works — busting uv.lock between passes and measuring the rebuild delta.

Now there’s a new optimization: move mai_kernels CUDA compilation into its own early Docker layer, before the uv sync steps. The idea is the same as the lockfile trick — separate the thing that rarely changes from the thing that changes all the time. Kernel source code is touched maybe once a month. uv.lock changes every few days. If kernels live in an earlier layer, a lockfile change doesn’t force recompilation.

Simple enough. Except proving it works almost broke my brain.

The optimization, in three lines

Before:

COPY uv.lock pyproject.toml ./
RUN uv sync --frozen                    # installs everything, including mai_kernels
                                        # mai_kernels compiles CUDA code during install

After:

# Early layer: compile mai_kernels (rarely changes)
COPY mai_kernels/ ./mai_kernels/
RUN pip install ./mai_kernels           # CUDA compilation happens here

# Later layer: install everything else (changes often)
COPY uv.lock pyproject.toml ./
RUN --env DISABLE_BUILD_EXT=1 \
    uv sync --frozen                    # skips mai_kernels rebuild

The DISABLE_BUILD_EXT=1 flag tells uv sync not to recompile extensions that are already installed. Since mai_kernels was compiled in the earlier layer, uv sync sees it’s present and skips it. When uv.lock changes, the early layer still cache-hits (kernel source unchanged), and only the uv sync layer rebuilds — without recompiling CUDA code.

That’s the theory. Now prove it.

The test that lied (by omission)

I ran the existing A/B test workflow. It busts uv.lock between passes, measures rebuild times. Baseline vs. optimized.

The numbers came back within ~10% of each other.

Huh.

The optimization is supposed to skip an entire CUDA compilation step. That should be a big delta — 10+ minutes of compilation, gone. A 10% difference is noise, not signal. Either the optimization doesn’t work, or the test is measuring the wrong thing.

I stared at the logs for a while. Then I saw it.

ccache: cache hit (direct)
ccache: cache hit (direct)
ccache: cache hit (direct)
... (400 more lines of cache hits)

Oh. Oh.

The baseline was also skipping CUDA compilation. Not because of the Docker layer optimization — because of ccache. The compiler cache was warm from the initial warm-build pass, so when the baseline “recompiled” mai_kernels after the uv.lock bust, ccache turned a 15-minute compile into a 30-second cache lookup.

My A/B test was comparing “skip compilation via Docker layer cache” against “skip compilation via ccache.” Both approaches skip it. Just through different mechanisms. The delta between “didn’t compile” and “also didn’t compile” is… not very big.

Caches hiding caches

Here’s what was happening in each pass:

Baseline (no early layer):

1. uv.lock busted → Docker rebuilds the uv sync layer
2. uv sync reinstalls mai_kernels → triggers CUDA compilation
3. ccache is warm from Pass 0 → compilation takes 30 seconds instead of 15 minutes
4. Total rebuild: ~3 minutes

Optimized (with early layer):

1. uv.lock busted → Docker skips the early mai_kernels layer (source unchanged)
2. Docker rebuilds only the uv sync layer
3. DISABLE_BUILD_EXT=1 → uv sync doesn’t even attempt compilation
4. Total rebuild: ~2.5 minutes

Delta: 30 seconds. Barely distinguishable from noise.

The optimization works perfectly. The test just can’t see it because ccache is also working perfectly. Two caches doing the same job, making the test look like a shrug.

This is the measurement equivalent of testing whether your umbrella works on a day when it’s not raining.

The solution: make it rain

If ccache is masking the delta, remove ccache. Not from the Dockerfile — from the test.

I added a bust_mode parameter to the A/B test workflow:

The uv_lock mode is what we already had — useful for measuring the uv sync optimization in isolation, and for real-world scenarios where ccache is typically warm.

The uv_lock_cold mode is the new one. It simulates the case that actually matters: someone changed uv.lock, and the CI runner doesn’t have a warm ccache. Fresh runner, cold caches, uv.lock changed — now what?

How to clear ccache inside BuildKit

This was trickier than it sounds. The ccache data lives in two places:

1. GitHub Actions cache — the ccache-cache/ directory restored by actions/cache
2. BuildKit cache mounts — the --mount=type=cache,target=/root/.cache/ccache volumes inside the builder

Clearing just one isn’t enough. If you delete the Actions cache but BuildKit still has the mount, ccache still hits. If you prune BuildKit but the Actions cache restores it on the next step, same result.

You need to nuke both:

if [ "$BUST_MODE" = "uv_lock_cold" ]; then
  echo "Clearing ccache for cold-compile scenario..."
  # Kill the Actions cache directory
  rm -rf ccache-cache/*
  # Kill BuildKit's cache mounts
  docker buildx prune --filter type=exec.cachemount -f 2>/dev/null || true
fi

This runs between Pass 0 (warm build) and the rebuild passes. Pass 0 still uses ccache normally — it primes the Docker layer cache so we’re measuring rebuilds, not cold starts. Then we yank the ccache rug out from under the baseline before the actual measurement begins.

The optimized branch doesn’t care. Its mai_kernels layer cached from Pass 0 is a Docker layer cache hit, not a ccache hit. Clearing ccache doesn’t affect it. The early layer was built once, it’s sitting in BuildKit’s layer store, and it’ll be reused as long as the kernel source files haven’t changed.

That asymmetry is the whole point. The optimization replaces ccache with layer caching for this specific step. The cold-ccache test reveals whether that replacement actually works.

The test matrix

Two bust modes × two questions:

The uv_lock run is still useful — it’s the common case in production, and it proves the optimization doesn’t regress anything. But uv_lock_cold is the smoking gun. It’s the scenario where the optimization earns its keep.

Think of it this way: uv_lock measures the floor (how much does the optimization help when everything’s going well already). uv_lock_cold measures the ceiling (how much does it help when things go wrong — cache miss, fresh runner, bad day).

Both numbers matter. But if you only measure the floor, you’ll conclude the optimization is a nothingburger. And you’d be wrong.

The meta-lesson

When you’re testing an optimization, you’re not just testing “does the optimization work.” You’re testing “does the optimization work net of everything else that’s also trying to solve the same problem.”

Every cache in your build pipeline is a potential confounder:

If you’re testing whether Layer Optimization X saves compilation time, and Cache Y also saves compilation time, your test is measuring max(X, Y) — not X. To isolate X, you need to disable Y. Not in production — just in the test.

This is why A/B testing infrastructure work is harder than A/B testing product features. Product A/B tests have one variable: the feature flag. Infrastructure A/B tests have N variables, and they’re all caches, and they all interact, and some of them are literally designed to make other optimizations invisible.

The umbrella principle

You can’t test an umbrella on a sunny day.

More precisely: you can, and you’ll conclude it doesn’t do anything. Which is true — on that day. The umbrella’s value is conditional on rain. If your test never produces rain, the umbrella looks useless.

Same with layer optimizations. The optimization’s value is conditional on ccache being cold. If your test never clears ccache, the optimization looks like it does nothing. It’s not that the optimization is broken — it’s that the test conditions don’t trigger the scenario where the optimization matters.

The fix is always the same: make sure your test includes the failure mode you’re optimizing for. If the optimization is “skip compilation when the layer is cached,” your test needs a scenario where compilation actually happens in the baseline. Otherwise you’re comparing two things that both skip compilation and congratulating yourself on a tie.

Where this fits in the series

Post 1: Split the lockfile to avoid unnecessary reinstalls.

Post 2: Build an A/B test to prove the split works.

Post 3 (this one): When the A/B test says “no difference,” check whether another cache is doing the optimization’s job. Add a test mode that disables the confounding cache so you can see the real delta.

The pattern generalizes. Every time you add a layer optimization to a build pipeline, ask: “What other cache could make this look like it does nothing?” Then build a test that disables that cache. You’ll need it eventually — either to prove the optimization works, or to debug why it stopped working.

Caches are great at their job. Sometimes too great. When your optimization’s best evidence is a tie, it might not be because the optimization failed. It might be because something else already succeeded.