Docker Builds: The Three Layers of Caching in CI

Our Docker image build has three independent caching systems. They have different lifetimes, different storage backends, different invalidation triggers. When people say “the build cache is warm” or “the cache missed,” they could be talking about any of the three — and the fix is completely different depending on which one broke.

Most CI debugging pain comes from not knowing which cache layer failed. Here’s the map.

Layer 1: GitHub Actions Cache (ccache)

The first cache lives in GitHub’s infrastructure. It stores compiled C++ object files — specifically, the .o files from CUDA kernel compilation.

Compiling CUDA kernels from source takes 10-20 minutes. The same source files produce the same .o files, so if the source hasn’t changed, there’s no reason to recompile. ccache handles this: it hashes the preprocessed source, checks if a matching object file exists in the cache, and returns it if so.

The cache key:

key: ccache-v3-${{ hashFiles('mai_kernels/**/*.cu', 'mai_kernels/**/*.cuh', 'mai_kernels/**/*.cpp') }}

When any .cu, .cuh, or .cpp file in mai_kernels changes, the hash changes, the cache key changes, and the whole ccache is busted. When none of them change, the cache hits and CUDA compilation drops from 15 minutes to under a minute.

Lifetime: Persistent across CI runs, stored in GitHub’s cache infrastructure. 10GB limit per repository — and we’re a monorepo, so that 10GB is shared with every other workflow that uses actions/cache.

Failure mode: Cache eviction due to the 10GB limit. GitHub evicts least-recently-used entries. If a week of main branch builds pushes enough other caches to fill the budget, your ccache entry gets evicted and the next build recompiles everything.

Layer 2: The Cache Dance

This is the weird one. It exists because of a fundamental problem: GitHub Actions cache lives on the runner filesystem, but Docker builds happen inside a container. They can’t see each other.

Think about it. The runner downloads the ccache from GitHub. It’s now sitting at some path on the runner machine. But the Dockerfile runs make inside a RUN instruction, which executes inside a Docker build container. That container has its own filesystem. It can’t see the runner’s filesystem. The ccache might as well be on the moon.

Enter buildkit-cache-dance — a GitHub Action that smuggles cache data from the runner filesystem into the Docker build context.

The mechanism:

1. GitHub Actions cache restores ccache files to a path on the runner
2. The Dockerfile uses RUN --mount=type=cache,target=/root/.cache/ccache — a BuildKit cache mount
3. buildkit-cache-dance intercepts this: before the build, it injects the runner’s cached files into BuildKit’s cache mount storage
4. During the build, RUN --mount=type=cache sees the injected files as if they were already cached
5. After the build, the dance extracts any new/changed files from the cache mount back to the runner filesystem
6. GitHub Actions cache saves the updated files

The configuration is a single JSON mapping:

{
  "ccache-cache": "/root/.cache/ccache"
}

Left side: the identifier for the runner-side cache. Right side: the path inside the container where BuildKit mounts it.

What this actually does: It’s a bidirectional sync between two filesystems that aren’t supposed to be able to talk to each other. The “dance” is reading BuildKit’s internal cache storage format, transplanting files in, letting the build run, and transplanting files back out.

Lifetime: Coupled to Layer 1 — it’s just a transport mechanism. If the GitHub Actions cache is warm, the dance has something to smuggle. If the cache is cold, the dance runs but there’s nothing to inject.

Failure mode: BuildKit version mismatch. The cache dance relies on the internal structure of BuildKit’s cache mount storage. If BuildKit changes how it stores cache mounts, the dance can silently fail to inject — the build proceeds without cached files, and you get a 15-minute CUDA compilation with no obvious error message.

Layer 3: Docker Layer Cache

This is the cache most people think of when they hear “Docker cache.” A Dockerfile is a sequence of layers. If a layer’s inputs haven’t changed, Docker reuses the cached output instead of re-executing the RUN command.

In CI, this cache has two sub-layers:

3a: Registry cache (persistent). After a successful build, we push the layer cache to ACR:

inf5acr.azurecr.io/yolo/pretraining/cache:main

Next build pulls this cache from the registry. If the Dockerfile hasn’t changed much, most layers hit the cache and only the changed layers rebuild.

cache-from: type=registry,ref=inf5acr.azurecr.io/yolo/pretraining/cache:main
cache-to: type=registry,ref=inf5acr.azurecr.io/yolo/pretraining/cache:main,mode=max

3b: Local cache (ephemeral). During a build, BuildKit maintains a local cache at $HOME/.buildx-cache. This helps when a single bake file builds multiple targets — layers shared between targets don’t need to be rebuilt. But since the runner pod is ephemeral, this cache dies with the job. It only helps within a single build, not across builds.

mode=min vs mode=max

That mode=max in the cache-to config matters a lot.

mode=min: Only caches the layers that appear in the final image. Intermediate build stages — the ones that compile libraries, run tests, download dependencies — are not cached.

mode=max: Caches all layers, including intermediate build stages.

For a typical application image, mode=min is fine. Your final image is what you deploy, and caching it covers most of what you need.

For our image, mode=min is catastrophic. The expensive work — compiling Flash Attention, Triton, MMCV, building CUDA kernels — happens in intermediate stages. The final image just copies the built artifacts. With mode=min, none of that compilation gets cached. Every build recompiles everything from source.

With mode=max, those intermediate compilation layers are cached in the registry. A build where only the Python deps changed skips the 25-minute CUDA compilation entirely because the intermediate layers are still valid.

The cost: mode=max pushes significantly more data to the registry. For us, that’s roughly 30-40GB of cache data per build. Registry storage is cheap. 25 minutes of CI compute on a GPU runner is not.

The three-container surprise

One more thing that trips people up. When you use the docker-container BuildKit driver (which is necessary for multi-platform builds and advanced cache export), a single build job runs three containers:

1. The runner container — the GitHub Actions runner itself
2. The DinD sidecar — the Docker daemon that the runner talks to
3. buildx_buildkit_yolo-builder0 — the BuildKit daemon that actually executes the build

BuildKit runs with its own configuration:

[worker.oci]
  max-parallelism = 32

[grpc]
  max-recv-message-size = 16777216  # 16MB

32 parallel layer builds. 16MB max gRPC message size for cache operations. These aren’t default values — they’re tuned for our specific build, which has many independent compilation steps that can run in parallel.

If you’re debugging a build and you exec into the runner container, you won’t see the build happening. It’s happening in the BuildKit container. If you’re looking at resource usage for the DinD sidecar, you’re looking at the wrong process. The heavy work is in buildx_buildkit_yolo-builder0.

How they interact

Here’s the full picture of a warm build:

1. GitHub Actions cache restores ccache .o files to runner filesystem (Layer 1)
2. buildkit-cache-dance injects those files into BuildKit’s cache mount (Layer 2)
3. BuildKit checks the registry cache for matching layers (Layer 3a)
4. For layers that need to rebuild — e.g., the RUN make step — BuildKit’s --mount=type=cache provides the ccache files, so compilation is incremental rather than from scratch (Layers 1+2 feeding into Layer 3)
5. Built layers are stored in local cache for use by subsequent bake targets (Layer 3b)
6. After build completes, updated cache layers pushed to registry (Layer 3a)
7. buildkit-cache-dance extracts updated ccache files back to runner (Layer 2)
8. GitHub Actions cache saves updated ccache files for next run (Layer 1)

A cold build — new runner, empty caches everywhere — skips steps 1-3 and does everything from scratch. That’s your 40-minute build.

A fully warm build takes 2-3 minutes. Same Dockerfile, same image, 95% time reduction.

The difference between “CI takes 3 minutes” and “CI takes 40 minutes” is whether three independent caching systems all hit. Miss any one of them and the corresponding build step goes cold. Miss all three and you’re compiling CUDA kernels on a machine that’s never seen your code before.

Debugging cache misses

When a build is unexpectedly slow, the question isn’t “did the cache miss?” It’s “which cache missed?”

The last row is important. Sometimes the build is slow because the build should be slow. If you changed Flash Attention’s commit pin, 15 minutes of recompilation is correct behavior. The cache is working — there’s just nothing to cache.

Three caches, three failure modes, three different fixes. Know which one you’re looking at before you start optimizing.