We spent weeks optimizing a Docker image build for ML training. Shaved 6+ minutes off dependency installs. Proved it with controlled A/B tests. Shipped it. Celebrated. And then we discovered that even with perfect caching, it still takes ~9 minutes to start building — because the cache itself has to be downloaded onto a machine that has never seen it before.
This is the story of the next bottleneck. And to tell it properly, I need to explain some things about computers that most people never think about.
What is a Docker image, actually
Imagine you’re a chef, and you’ve perfected a recipe. It involves a specific oven, specific pots, specific spice rack, specific knife set — a whole kitchen setup that took you years to dial in. Now you need to cook this same meal in 50 different kitchens around the world.
You have two options. You can walk into each kitchen and spend hours setting everything up from scratch. Or you can photograph your entire kitchen — every shelf, every drawer, every setting — pack those photographs into a shipping crate, and send it ahead. When you arrive, someone has already recreated your kitchen exactly. You just start cooking.
That’s Docker. A Docker “image” is a portable snapshot of an entire computing environment — the operating system, the software, the libraries, the configurations, all of it. You build it once, and then you can recreate that exact environment on any machine in seconds. The alternative is manually installing everything from scratch every time, which is exactly as tedious and error-prone as it sounds.
Our team trains large language models. The Docker image for that includes Python, CUDA libraries for GPU computation, machine learning frameworks like PyTorch, and hundreds of supporting packages. It’s about 66 gigabytes. Building it from scratch takes 40 minutes. So we really don’t want to rebuild it from scratch.
Layers: the transparent sheet trick
Docker images aren’t monolithic blobs. They’re built in layers — thin, stackable slices, each representing one step of the setup process.
Think of it like making a sandwich. Layer 1: bread. Layer 2: lettuce. Layer 3: tomato. Layer 4: cheese. Layer 5: more bread. Each layer sits on top of the previous one.
Now here’s the clever part. Docker keeps each layer separately and labels it with a fingerprint of its contents. If you make the exact same sandwich tomorrow — same bread, same lettuce — Docker recognizes the layers haven’t changed and reuses them. It doesn’t re-slice the tomato. This is called layer caching, and it’s the reason Docker builds can be fast even when the image is huge.
But there’s a catch. The layers are a stack. If you change the lettuce (layer 2), Docker has to redo the lettuce, the tomato, the cheese, and the top bread — everything above the change. It can’t tell that the tomato didn’t depend on the lettuce. It just sees “something changed in the middle” and replays the rest.
This is why layer ordering matters so much. You want the things that never change at the bottom and the things that change often at the top. We optimized our layers to follow this principle. The two-phase trick was specifically about keeping a stable layer of Python dependencies at the bottom so that small dependency bumps only touch a thin layer at the top.
That optimization worked beautifully. Phase 1 (the stable bulk of dependencies) stays cached across builds. Phase 2 (the tiny diff) takes under a second. Rebuild time dropped from 11 minutes to 90 seconds.
So where does the 9 minutes come from?
The rental car problem
Here’s a concept that seems obvious once you hear it but trips up everyone the first time: the machines that build our Docker images don’t remember anything.
In our CI (continuous integration — the system that automatically builds and tests code every time someone pushes a change), the builds run on virtual machines. A virtual machine, or VM, is a computer that exists only in software — a fake computer running inside a real computer. Cloud providers like Azure spin up thousands of these. You rent one for an hour, run your stuff, and it evaporates.
The key word is evaporates. Each build gets a brand new VM. Fresh operating system. Empty hard drive. No memory of any previous build. It’s like renting a car every day and discovering the trunk is always empty — no matter how much stuff you left in yesterday’s car, today’s car is a stranger.
This means the Docker layer cache — all those carefully optimized layers we spent weeks perfecting — doesn’t exist on the machine when the build starts. The machine has never seen our image before. So before Docker can check “hey, has this layer changed?”, it first has to download the previous version of every layer from somewhere.
That “somewhere” is a registry — think of it as a warehouse where you store your pre-built images. Ours lives in Azure Container Registry (ACR). When a build starts, Docker phones the registry and says: “Send me what you had last time so I can figure out what’s changed.”
For a 66GB image, “send me what you had” is not a trivial request. Even compressed, that’s a lot of data to download over a network. And that download currently takes about 9 minutes.
Nine minutes. Before a single line of our build logic runs.
Why this wasn’t a problem before
Because everything else was slower.
When the uv sync step (Python dependency installation) took 20 minutes, nobody noticed that cache download took 9. The 20-minute step dominated the timeline. Fix the 20-minute step — which we did — and suddenly the 9-minute step is the new ceiling.
This is a pattern so common it has a name: Amdahl’s Law. When you speed up one part of a system, the other parts become the bottleneck. Optimize long enough and you’ll eventually be fighting something you didn’t even know was there.
Masatoshi — a teammate who works on image build infrastructure — is the one who surfaced this. He’d been digging into build timelines and noticed that even perfectly cached builds (where Docker reuses every layer, does zero actual work) still take ~9 minutes. All of that time is spent downloading cached layers from the registry onto the fresh runner.
So the question became: can we keep a local copy of the cache on the build machine, so Docker doesn’t have to download it every time?
The obvious fix and its three problems
The obvious fix is persistence. Don’t throw away the build machine after every use. Keep it around. Let the Docker cache live on local storage. Next build on the same machine? Cache is already there. Zero download time.
Cloud VMs can technically be persistent — you just don’t destroy them between jobs. The practical version is: use a VM with a fast local disk, store the Docker cache on that disk, and reuse the VM across builds.
This is where Masatoshi identified three hard problems.
Problem 1: The ARM architecture gap
Here I need to explain something about how computers work at the lowest level.
Every computer has a processor — the chip that actually runs instructions. Processors come in different architectures, which is a fancy way of saying they speak different languages. The two that matter here are AMD64 (also called x86_64) and ARM64 (also called aarch64).
AMD64 is the architecture in most desktops and servers. It’s what you’re probably reading this on if you’re using a PC. ARM64 is what’s in your phone, in Apple’s M-series Macs, and increasingly in cloud servers because it uses less power.
We need Docker images for both architectures. Our training jobs run on AMD64 GPU servers, but some of our infrastructure runs on ARM64. So we build two versions of every image.
Now, Azure offers a type of VM called L-series that comes with extremely fast local storage. Specifically, NVMe SSDs.
Time for an analogy. A regular hard drive is like a filing cabinet — you open a drawer, flip through folders, pull out a file. An SSD (solid-state drive) is like a bookshelf — everything is right there, grab what you need. An NVMe SSD is like having the book already open on your desk. It’s local storage that’s absurdly fast because it’s physically attached to the machine’s motherboard, not connected through a network cable or a bus.
The speed metric that matters here is IOPS — input/output operations per second. How many “grab a file” requests the drive can handle per second. NVMe SSDs can handle hundreds of thousands. A network-attached storage might handle tens of thousands. Docker’s layer cache is IO-heavy — it’s constantly reading and writing chunks of filesystem data. High IOPS = fast cache operations.
L-series VMs have NVMe SSDs. Perfect for Docker cache.
The problem: L-series VMs only come in AMD64. There is no ARM L-series.
Which means we can set up persistent Docker cache with fast local storage for AMD64 builds… but not for ARM64 builds.
The current workaround for ARM64 is cross-platform emulation. We build ARM64 images on an AMD64 machine using a tool called QEMU, which essentially translates ARM64 instructions into AMD64 instructions on the fly.
Imagine trying to cook French food using only Japanese kitchen tools. Technically possible — the fundamental ingredients and techniques translate — but every single step involves a moment of “okay, this whisk is shaped differently, how do I adapt?” It’s brutally slow.
How slow? We tried it on a 96-core machine with 384 gigabytes of RAM. That’s an enormous computer. Couldn’t finish the build in 5-6 hours. The same build takes 40 minutes on a native ARM64 machine.
So we need ARM64 VMs with fast local storage, and Azure doesn’t offer that combination. Masatoshi is exploring alternatives — E48ds_v5 VMs that have 1.8 terabytes of local storage (but it’s regular SSD, not NVMe), or Premium/Ultra SSDs (which are technically remote storage but with high IOPS guarantees). Whether those are fast enough is an open question.
Problem 2: The time machine cleanup problem
Even with persistent storage, you eventually run out of space. A 66GB image accumulates cache layers over time. You need to clean up old ones — prune them, in Docker terminology. Keep the recent stuff, throw away the stale stuff.
This should be simple: delete anything older than, say, a week.
It was not simple.
There was a build setting called SOURCE_DATE_EPOCH=0. This is a reproducibility feature — it forces all timestamps in the image to January 1st, 1970 (the Unix epoch, the arbitrary date that computers count time from). The idea is that builds done at different times produce byte-identical images, which makes it easier to verify that nothing sneaky changed between builds.
Noble goal. Terrible side effect. When every image has the same timestamp, you can’t tell which ones are old. Your pruning script looks at the cache and sees a thousand entries all claiming to be from 1970. It’s like trying to organize a filing cabinet where every folder is labeled “January 1st.”
Masatoshi fixed this by removing SOURCE_DATE_EPOCH=0 from the build config. Images now have real dates. Pruning by age works again.
Small fix. But you’d be amazed how many small fixes turn out to be prerequisites for the actual fix you wanted.
Problem 3: The cold start chicken-and-egg
Even with persistent VMs and working cleanup, there’s the first-build problem. The very first build on a fresh runner has no cache at all. Nothing on the local disk. Back to 40 minutes.
This matters because VMs do get replaced — OS updates, scaling events, hardware failures. When a new VM appears, it starts cold.
The discovery here was interesting. Docker’s build engine, BuildKit, has a feature: it can use the registry itself as a cache source. On cold start, instead of building from zero, Docker can pull cached layers from the registry — the same registry where we push finished images. This is called registry cache backend.
But there’s a requirement: the Docker instance needs to be using a specific storage driver called containerd image store. Without it, the registry cache feature doesn’t work. Masatoshi confirmed this is available and tested it — cold starts with registry cache are significantly faster than cold starts from nothing.
So the architecture becomes: fast local NVMe cache for warm builds (most builds), registry cache fallback for cold starts (rare but important). Belt and suspenders.
“Does the cache have to be local?”
Sriram — another teammate who thinks about infrastructure problems — asked the natural question: forget local storage entirely. What if the cache lives on a remote disk? Put it on a network-attached SSD, mount it from the VM, done. No persistence problem because the disk outlives the VM.
Masatoshi’s answer was illuminating.
Premium SSD and Ultra SSD in Azure are technically remote storage — they’re not physically bolted to the machine. They connect over the network. But BuildKit still has to load cached layers into local memory to reconstruct the filesystem. The bytes have to physically travel from the storage device, through the network, into the VM’s RAM.
This is the fundamental constraint. Caching isn’t magic. “Cached” doesn’t mean “free.” It means “already computed, but still needs to be loaded.” The question is always: how fast can you load it?
With local NVMe, the data travels a few centimeters across a PCIe bus. Hundreds of thousands of IOPS. Microseconds per operation.
With a network-attached SSD, the data travels through a network switch, across a datacenter fabric, into the VM. Maybe tens of thousands of IOPS. Milliseconds per operation.
Is the difference significant for a 66GB image cache? That’s what we’re trying to find out. The answer probably depends on the specific access pattern — is Docker reading a few large blobs (network bandwidth matters more) or thousands of small files (IOPS matters more)?
Two hypotheses about layer splitting
While Masatoshi was working the VM and storage angle, I had a different idea. What if the problem isn’t where the cache lives, but how big the cached chunks are?
Our image cache is structured as a handful of very large layers — one layer for CUDA libraries (~10GB), one for Python packages (~5GB), etc. When Docker downloads the cache, it pulls these massive blobs sequentially. One at a time. One 5GB download, then the next.
Hypothesis 1: Parallelism. What if we split large layers into many smaller layers? Docker can download multiple layers simultaneously. Instead of one 5GB download taking 3 minutes, maybe ten 500MB downloads running in parallel take 30 seconds. Same total data, but utilizing more of the network bandwidth at once.
Sriram was skeptical. “Are you sure the bottleneck is network bandwidth?” he asked. “What if it’s disk IOPS on the runner? Downloading ten things in parallel doesn’t help if the disk can’t write ten things in parallel.”
Fair point. If the bottleneck is the speed of the storage device on the runner — how fast it can write incoming data to disk — then parallelizing the downloads just creates a traffic jam at the other end. Like having ten delivery trucks arrive at a building with one loading dock.
Hypothesis 2: Cache granularity. Smaller layers mean more granular cache invalidation. Right now, if anything in the Python dependencies layer changes, the entire 5GB layer is re-downloaded. With finer-grained layers — maybe one per major dependency — you’d only re-download the specific sub-layer that changed.
Sriram liked this one better. But he noted: you’d need to sort the sub-layers by how frequently they change. Put the volatile stuff (dependencies that get bumped weekly) in separate layers from the stable stuff (dependencies that haven’t changed in months). Otherwise you’re doing the same cache invalidation dance, just with more layers.
This is the same principle as the two-phase lockfile trick, just applied at a different level. That trick separates stable dependencies from volatile dependencies in the lockfile. This would separate them at the Docker layer level.
“Measure before you theorize”
Sriram’s best suggestion was the most boring one: stop guessing and start measuring.
“Can you get CPU utilization and IOPS metrics from the build runners?” he asked. “If CPU is pegged during cache download, that’s a decompression bottleneck. If IOPS is maxed but CPU is idle, that’s a storage bottleneck. If neither is saturated but things are still slow, it’s probably network latency.”
This is the kind of advice that’s obvious in retrospect and completely non-obvious when you’re deep in hypotheses about layer splitting. You can debate architecture all day, but the profiler doesn’t lie.
We use Datadog for CI observability. The pipeline data is already flowing — job durations, success rates, that kind of thing. But we don’t currently have system metrics from the runners themselves during builds. No CPU, no disk IOPS, no memory pressure.
Adding that instrumentation is the next step. Once we can see what the machine is actually doing during those 9 minutes of cache download, the right optimization will probably be obvious.
Where this stands
Here’s the situation as of today:
What we know:
- Cache download from registry takes ~9 minutes on ephemeral runners
- This is now the dominant bottleneck after uv sync optimization
- Persistent local cache on NVMe SSDs would eliminate this, but ARM64 L-series VMs don’t exist
- Timestamp cleanup is fixed, enabling cache pruning on persistent VMs
- Cold start has a solution via containerd image store + registry cache backend
What we’re exploring:
- E48ds_v5 VMs (1.8TB local SSD, non-NVMe) — are they fast enough?
- Premium/Ultra SSD (remote, high IOPS) — is the latency acceptable for Docker’s access pattern?
- Layer splitting — would it help, and is the bottleneck even where we think it is?
What we need:
- System-level metrics from build runners during cache operations (CPU, IOPS, network)
- Real-world testing of alternative VM SKUs with Docker cache workloads
- A decision on whether to invest in persistent runners vs optimizing the registry cache path
This post doesn’t have a clean ending because the work doesn’t have one yet. We peeled one layer of the onion (uv sync: fixed), found another layer underneath (cache download: not fixed), and now we’re squinting at the next layer trying to figure out what kind of onion this is.
The pattern is always the same. Fix the loudest bottleneck. Discover the quieter one that was hiding behind it. Measure. Repeat. The build gets faster not because of one brilliant idea, but because someone keeps asking “okay, but why is this part slow?”