Docker Builds: Anatomy of a 40-Minute Build

Our pretraining image build takes 40 minutes on a cold run. Here’s where that time actually goes.

Two stages

The build has two Dockerfiles chained through a bake file. Stage one builds the base — system packages, CUDA libraries compiled from source, tools. Stage two installs Python deps and compiles our CUDA kernels on top.

shared (docker/Dockerfile)           → 25-40 min
  └── pretraining (pretraining/Dockerfile)  → 8-15 min

Stage one: the CUDA gauntlet

The shared Dockerfile compiles four major libraries from source. All sequential — each waits for the one before it:

Flash Attention 3 — 10-20 min
Triton — 5-10 min
MMCV — 5-10 min
DeepEP — 5-10 min

Plus Redis, NCCL, apt packages, and tooling — another 3-5 minutes of overhead.

These are sequential because Docker layers are sequential. Layer N+1 can’t start until layer N finishes. If the Flash Attention commit pin changes, Triton, MMCV, and DeepEP all rebuild too — even though they didn’t change. Docker doesn’t know they’re independent.

The layer ordering problem

Docker’s layer cache is a stack. Change something in the middle and everything above it rebuilds. The current ordering doesn’t account for this — things that change frequently (like commit pins for FA3 or DeepEP) sit in the middle of the stack, forcing full rebuilds of layers that follow.

Optimal ordering: least-changing layers first, most-changing last. apt packages and system tools almost never change — they should be at the top. Source compilations with pinned commit hashes change occasionally — they should be near the bottom, ordered so that each one’s rebuild doesn’t cascade into others.

Stage two: uv sync

The pretraining Dockerfile does two uv sync calls:

# Install external deps (bust by uv.lock changes)
RUN --mount=type=cache,target=/root/.cache/uv \
    uv sync --no-install-workspace --frozen --package mai_job

# Install workspace packages + compile CUDA kernels
RUN --mount=type=cache,target=/root/.cache/ccache \
    uv sync --package mai_job --frozen

The first installs external Python packages — 3-5 minutes when cold. The second compiles mai_kernels (custom CUDA code) — 5-10 minutes depending on ccache warmth.

This split is actually well-designed. Deps and code compilation are in separate layers, so a code change doesn’t re-download all of PyTorch. The --mount=type=cache for uv and ccache is smart — when the caches are warm, this whole stage takes 1-2 minutes.

When they’re cold? 15 minutes.

The disk constraint

One more thing. The runner’s Docker disk is 1.2TB, and it’s 99% full. Each image is ~66GB. The builder barely has room for intermediate layers during a build, and Docker’s garbage collector might evict the caches that make warm builds fast.