Someone on our team noticed that GPU test jobs were taking 17 minutes just to pull the container image. Not run the tests. Not compile anything. Just downloading the image so the test could start.
The working theory was “big image, slow pull.” The image is 40GB, so sure, that tracks. But other jobs pulling the same image were instant. Same image, same cluster, same network. One job pulls in seconds, another takes 17 minutes.
The difference isn’t the image. It’s who’s doing the pulling.
Two ways to run a container in CI
GitHub Actions Runner Controller (ARC) supports two modes for running containerized jobs on Kubernetes: Docker-in-Docker (DinD) and Kubernetes mode.
DinD mode: The runner pod gets a sidecar container running a full Docker daemon. When your workflow says container: my-image:latest, the runner tells its local Docker daemon to pull and run the image.
Kubernetes mode: The runner pod asks the kubelet — the Kubernetes node agent — to create a separate pod for the job. The kubelet pulls and runs the image directly, just like it would for any other K8s workload.
Same end result from the workflow’s perspective. Wildly different plumbing underneath.
The cache problem
Here’s the table that explains everything:
| Aspect | DinD | K8s Mode |
|---|---|---|
| Who creates the test container? | Docker daemon (inside pod) | Kubelet (node level) |
| Who pulls the image? | Docker daemon | Kubelet |
| Image cache location | Inside ephemeral Docker daemon | Node-level kubelet cache |
| Cache persists after job? | No — pod dies, cache lost | Yes — stays on node disk |
| Second pull of same image? | Full pull from registry | Instant (already cached) |
In DinD mode, the Docker daemon lives inside the runner pod. Its image cache lives inside the Docker daemon. When the pod gets cleaned up after the job finishes — which is what ephemeral runners do — the Docker daemon disappears and takes its cache with it.
Next job arrives on the same node. New pod, new Docker daemon, empty cache. Pull the entire 40GB image again from scratch.
In Kubernetes mode, the kubelet pulls the image to the node’s disk. The kubelet is a node-level process — it doesn’t care about individual pods coming and going. The image stays cached on the node. Second job on the same node? Image is already there. Pull takes zero seconds.
The car rental analogy
DinD is like renting a car with an empty trunk every single time. You need your 40GB of luggage to get to work, so you drive to the storage facility, load the trunk, drive to the office, and when you return the car, they empty the trunk. Tomorrow? Same drive. Same loading. Same 17 minutes.
Kubernetes mode is like having a locker at the airport. You drop your luggage once, and it’s there every time you fly. The locker doesn’t care which plane you’re on or which airline you booked. It’s attached to the building, not the flight.
Why DinD cache is ephemeral
You might think “well, just mount the Docker cache to a persistent volume.” That’s a reasonable instinct. Let’s look at the actual Helm values for our GPU DinD runners, from values-gpu-h100-dind.yaml:
volumes:
- name: docker
emptyDir: {}
- name: work
emptyDir: {}
emptyDir: {}. That’s a Kubernetes volume type that says “give me a fresh empty directory when the pod starts, and throw it away when the pod dies.” No hostPath, no persistent volume claim, no NFS mount. Just an ephemeral scratch space.
The Docker daemon inside the sidecar writes to this emptyDir. All layers, all image data, all cache — sits on ephemeral storage that exists for exactly the lifetime of one CI job.
You could change this to a hostPath volume pointing to something like /var/lib/docker-cache on the node. But that introduces a new set of problems: cache invalidation across daemon versions, garbage collection when the disk fills up, cache corruption when two Docker daemons try to share the same storage, security implications of exposing host filesystem to runner pods. The Kubernetes community generally recommends against sharing Docker storage across pods for good reasons.
What the test data showed
We ran an experiment (PR #16791) to compare the two modes head-to-head. Same GPU tests, same images, same cluster. The results were… surprising.
Both modes averaged about 4.5 minutes per job.
No 17-minute pulls in either mode during the test window. Not what we expected.
There are a few explanations for this. The test ran on a relatively quiet cluster, so nodes had warm caches. The image hadn’t changed recently, so even DinD might have been pulling from a local registry mirror. And the sample size was small — edge cases like cold nodes or registry throttling didn’t show up.
The 17-minute pulls are real — we see them in production metrics. But they’re intermittent, which makes them harder to catch in a controlled test. They happen when a new node scales up (cold cache everywhere), when the image changes and all caches invalidate simultaneously, or when the registry is under load from multiple concurrent pulls.
The legacy baggage
So if K8s mode has better caching, why aren’t we using it everywhere?
History. The older version of ARC (before Runner Scale Sets) had stability issues with Kubernetes mode. Pods would get stuck, cleanup would fail, jobs would hang indefinitely. DinD was the reliable option. The team switched to DinD, the problems went away, and nobody revisited the decision.
Runner Scale Sets — the newer ARC architecture — is supposed to have fixed the K8s mode stability issues. It uses a different pod lifecycle management approach. But “supposed to have fixed” and “we’ve verified it works in our specific cluster configuration with our specific GPU scheduling constraints” are two very different statements.
The migration path exists. The risk tolerance to test it hasn’t caught up yet.
Three layers of infrastructure
One thing that makes CI infrastructure debugging uniquely disorienting is that there are three independent layers, and a problem at any layer can produce symptoms that look like they belong to a different layer.
Layer 1: The workflow. This is the .github/workflows/*.yml file. It declares what to build and test. “Run pytest on these packages with this container image.” It doesn’t know or care about Kubernetes, Docker daemons, or node pools.
Layer 2: The runner configuration. This is the Helm chart that configures ARC — values-gpu-h100-dind.yaml or values-gpu-h100-k8s.yaml. It declares where and how jobs run. DinD vs K8s mode, resource requests, volume mounts, GPU scheduling. It doesn’t know what tests you’re running.
Layer 3: Terraform. This declares what machines exist. Node pool sizes, GPU types, spot vs on-demand, autoscaling policies. It doesn’t know anything about GitHub Actions or Docker.
A 17-minute image pull looks like a Layer 1 problem (“our CI is slow”). But it’s actually a Layer 2 problem (DinD mode + ephemeral volumes = no cache). And it might be exacerbated by a Layer 3 problem (autoscaler spinning up a cold node with nothing cached).
Debugging CI requires holding all three layers in your head simultaneously. Most people only think about Layer 1 because that’s the file they edit. The other two layers are “someone else’s problem” until a 40GB image pull reminds you they exist.
The actual fix
There are three paths forward, in increasing order of ambition:
Option 1: Image pre-pulling. Run a DaemonSet that pulls the GPU test image to every node on a schedule. When a DinD job starts, the Docker daemon still does its own pull — but the registry serves it from a local mirror or the node’s containerd cache, depending on the runtime. Doesn’t fix the architecture, but masks the symptom.
Option 2: Switch to K8s mode on Runner Scale Sets. The principled fix. Kubelet caches at the node level, second pulls are instant. Requires validation that K8s mode is stable in the current ARC version, GPU scheduling works correctly, and pod cleanup is reliable. Medium effort, high payoff.
Option 3: Shrink the image. 40GB is a lot of image. If you can get it to 10GB, even a cold pull is tolerable. But there are reasons it’s 40GB — CUDA libraries, PyTorch, compiled kernels — and most of that isn’t optional for GPU tests.
Option 2 is the right answer. But Option 1 is the answer you ship this week while you’re validating Option 2.
The pattern generalizes: when your CI is slow, check who’s caching what and where that cache lives. If the cache is attached to something ephemeral, your first job will always be slow. Make the cache live somewhere that outlasts a single job, and the problem goes away. Whether that’s kubelet node cache, a persistent volume, or a local registry mirror — the principle is the same.
Don’t rent a car with an empty trunk every morning.