For a brief, glorious period, our Docker image builds took about one minute.
We had a persistent BuildKit daemon running in our Condor cluster. Runners connected to it remotely, sent the build context, and BuildKit — with its warm layer cache still intact from the last build — would spit out a fresh image almost instantly. Changed one Python file? Rebuild one layer. Done.
Then we turned it off.
How remote BuildKit works
In a normal CI Docker build, the BuildKit daemon is ephemeral. It starts with the job, runs the build, and dies when the pod is cleaned up. Layer cache? Gone. Next job starts from zero.
A remote BuildKit daemon flips this. The daemon runs as a persistent Kubernetes deployment. It stays alive between jobs. Its layer cache accumulates over time — every successful build warms the cache for the next one.
The runner connects to it using docker buildx create --driver remote:
docker buildx create \
--name yolo-builder \
--driver remote \
tcp://buildkit-service.namespace.svc.cluster.local:1234
Now when the runner runs docker buildx bake, it’s not building locally. It’s sending a build request to the remote daemon, which has all the layers from the last successful main branch build sitting in its local storage.
With warm layers, the build is just diffing. What changed since last time? One Python file? Rebuild that one layer, reuse everything else. The 25-minute CUDA compilation stage? Completely skipped — those layers haven’t changed.
One minute. Sometimes less.
The failure mode
The problem is the word “context.”
When you run docker buildx build, one of the first things that happens is the Docker client packages up the build context — everything the Dockerfile might need — and sends it to the BuildKit daemon. For a local daemon, this is a copy between processes on the same machine. Fast, free, doesn’t touch the network.
For a remote daemon, this is a network transfer.
Our repo is big. Not “oh it’s a few hundred megabytes” big. It’s a monorepo. The build context — even with a .dockerignore — is 2-5GB depending on what’s checked out.
One PR build: 2-5GB uploaded over the network to the remote BuildKit pod. Manageable.
Three concurrent PR builds: 6-15GB uploaded simultaneously.
Five concurrent builds during a busy afternoon: 10-25GB of network traffic, all hitting the same BuildKit pod, while the pod is also trying to execute builds, manage layer cache, and respond to health checks.
The BuildKit pod has finite CPU, finite memory, finite disk I/O, and finite network bandwidth. 25GB of concurrent uploads is not a “stress test.” It’s a denial of service.
How it actually dies
The failure cascade goes like this:
- Multiple CI runs start within a few minutes of each other (normal during active development)
- Each one begins uploading 2-5GB of build context to the remote BuildKit
- Network I/O on the BuildKit pod spikes
- BuildKit’s gRPC server starts queueing requests — it can’t process context uploads and build layers simultaneously at this rate
- Memory usage climbs as multiple contexts sit in buffers waiting to be processed
- One of three things happens: OOM kill, disk I/O timeout, or gRPC deadline exceeded
- The pod restarts
- All in-flight builds fail
- Kubernetes reschedules the pod, which starts with a cold cache
- The next build is a 40-minute cold build instead of a 1-minute warm build
The ironic part: the thing that makes remote BuildKit fast (persistent cache) is exactly what makes it fragile under concurrency. A single persistent daemon is a single point of failure, and the failure mode erases the benefit.
This is not DinD vs K8s mode
I want to be precise about this because these two topics get conflated constantly.
| Topic | What it’s about | The problem | Which layer |
|---|---|---|---|
| DinD vs K8s runner mode | Running GPU tests | Image pull time (17 min) | Runner infrastructure |
| Remote BuildKit | Building images | Context upload overwhelming builder | Build infrastructure |
Different jobs. Different infrastructure. Different failure modes.
DinD vs K8s mode is about how the runner pulls a pre-built image to run tests. The image already exists in a registry; the question is whether the pull cache is ephemeral (DinD) or persistent (kubelet).
Remote BuildKit is about how the image gets built in the first place. The Dockerfile needs to be executed, layers need to be compiled, and the result needs to be pushed to the registry.
You could use DinD runner mode with remote BuildKit. You could use K8s runner mode with local BuildKit. They’re orthogonal. The only connection is that they both involve Docker and they both affect CI latency.
The current state
Remote BuildKit is disabled. We’re using registry-based layer caching instead.
The tradeoff:
| Remote BuildKit | Registry Cache | |
|---|---|---|
| Warm build time | ~1 minute | 12-17 minutes |
| Cold build time | 40 minutes (same) | 40 minutes (same) |
| Concurrency handling | Collapses | Fine (each runner builds independently) |
| Stability | Fragile under load | Rock solid |
| Infrastructure cost | Persistent pod + storage | Registry storage |
Twelve minutes instead of one. That’s an 11-minute tax we pay on every build, every day, because the 1-minute solution can’t handle three people merging PRs within the same hour.
The numbers add up. If we build the image 10 times a day — not unreasonable for a monorepo with active development — that’s 110 extra minutes of CI compute daily. About 550 minutes a week. Nine hours of GPU runner time per week, spent rebuilding layers that haven’t changed.
Could it be fixed?
There are a few architectural approaches that could give us the speed of remote BuildKit without the concurrency collapse:
Sharding. Instead of one BuildKit pod, run N pods behind a load balancer. Each PR hashes to a consistent builder, so the same PR always hits the same cache. Concurrent builds go to different builders. The problem: N pods means N copies of the layer cache, and our cache is 30-40GB. At N=5, that’s 150-200GB of persistent storage for build cache. It also means each builder’s cache is warmer for some images and cold for others.
Queue-based serialization. Accept that only one build can run at a time on the remote builder. Queue concurrent builds and run them sequentially. The problem: queue depth during busy periods. If 5 PR builds queue up and each takes 1-2 minutes, the last one waits 8-10 minutes. Better than 17 minutes with registry cache, worse than 1 minute, and you’ve added a queue system to maintain.
Hybrid approach. Use remote BuildKit for the main branch build (which runs once per merge, rarely concurrent) and registry cache for PR builds (which are frequent and concurrent). Main branch builds are the ones that update the registry cache anyway, so this might give you the warm cache benefit where it matters most.
Context pruning. The root cause is 2-5GB context uploads. A better .dockerignore — or restructuring the Dockerfile to use multi-stage builds with targeted COPY instructions — could dramatically reduce context size. If context is 200MB instead of 2GB, the concurrency math changes entirely.
None of these are simple. Each trades one problem for another. The team made a pragmatic call: registry cache is 12 minutes slower but doesn’t page anyone at 2 AM. In infrastructure, “boring and reliable” almost always beats “fast and fragile.”
The lesson
Remote BuildKit is a local maximum. When you test it with one build, it’s incredible. The demo is amazing. You push a Python change and the image appears in the registry in 60 seconds. Everyone applauds.
Then Monday morning happens. Five PRs merge before standup. The builder pod OOMs. All builds fail. Someone manually retriggers them. They all start simultaneously. The pod OOMs again.
The mistake isn’t trying remote BuildKit. It’s evaluating it under synthetic conditions — one build, warm cache, no contention — and extrapolating to production. The real environment has concurrency, and concurrency changes everything.
Any CI optimization that works beautifully for one job but collapses under N concurrent jobs is not an optimization. It’s a demo.