A 40-minute Docker build has two problems: CUDA compilation that’s inherently slow, and cache busts that make it slow even when nothing meaningful changed. Here’s what actually fixes each one — without switching build systems or rewriting Dockerfiles from scratch.
Persist the uv cache
The easiest win. Our RUN lines already use --mount=type=cache,target=/root/.cache/uv, but on ephemeral runners or after Docker GC, that cache is empty.
Fix: use GitHub Actions’ cache action to save and restore /root/.cache/uv between workflow runs. Even when uv.lock changes, uv won’t re-download packages it already has. The 3-5 minute dep install drops to under a minute.
Same logic applies to ccache — persist it and the CUDA kernel compilation goes from 5-10 minutes to 1-2 minutes on incremental changes.
Effort: low. Savings: 2-5 min per build.
Pre-build CUDA wheels
The single biggest time sink is compiling Flash Attention, Triton, MMCV, and DeepEP from source. They compile inside the Docker build, every time their commit pin changes.
Alternative: build each one as a separate CI step, produce a .whl file, store it in ACR or blob storage, and pip install the wheel in the Dockerfile. No compilation during image build.
The wheel only needs to rebuild when its source commit changes. For something like MMCV that changes once a quarter, that’s a lot of compilation you’re skipping.
Effort: medium. Savings: 10-20 min per build when pins change.
Smart lockfile hashing
Right now, any change to uv.lock busts the dep install layer — even changes to packages that mai_job doesn’t depend on. That’s ~70% of the cache busts.
The fix: extract mai_job’s resolved dependency subset from uv.lock, hash just that, and use it as the Docker cache key. If your deps didn’t change, don’t rebuild.
This could be a small script that runs before the Docker build — parse uv.lock, filter to mai_job’s transitive deps, write a hash file, and COPY that instead of the full lockfile for cache-key purposes.
Effort: medium. Savings: eliminates most false cache busts.
Parallel compilation with multi-stage builds
The big architectural change. Right now FA3, Triton, MMCV, and DeepEP compile sequentially — each takes 5-20 minutes, and they can’t overlap.
Multi-stage Dockerfiles fix this:
FROM base AS build-fa3
RUN git clone ... && python setup.py bdist_wheel
FROM base AS build-triton
RUN git clone ... && pip wheel ./python
FROM base AS final
COPY --from=build-fa3 /wheels/*.whl /tmp/
COPY --from=build-triton /wheels/*.whl /tmp/
RUN pip install /tmp/*.whl
Docker BuildKit can build independent stages in parallel. Four 10-minute compilations in parallel instead of 40 minutes in series.
The catch: this requires the docker-container buildx driver, not the docker driver we currently use. The docker driver can’t parallelize stages. Switching means losing access to the local Docker image cache — the first build after switching will be fully cold. After that, registry-based caching (cache-from/cache-to pointing at ACR) takes over and subsequent builds are fast.
Effort: high. Savings: 15-30 min on cold builds.
What I’d actually do
In order:
- Persist uv + ccache — do it this week, easy, immediate payoff
- Pre-build CUDA wheels — next sprint, biggest single improvement
- Smart lockfile hashing — when you have a spare afternoon
- Parallel multi-stage — future project, requires driver migration
Steps 1 and 2 together take the build from 8-15 minutes to about 2-3 minutes on a warm cache. That’s the difference between “go get coffee” and “it’s already done.”