Our pretraining Docker image rebuilds its entire Python dependency layer 4 times a day. Not because our dependencies changed. Because someone else’s did.
One lockfile to rule them all
We use uv workspaces. One uv.lock for the whole monorepo — 20+ packages, dozens of teams. When anyone adds or bumps a dependency in any package, uv.lock changes.
The Dockerfile copies that lockfile early:
COPY ../../uv.lock /app/yolo/uv.lock
RUN uv sync --no-install-workspace --frozen --package mai_job # 3-5 min
RUN uv sync --package mai_job --frozen # 5-10 min (CUDA)
When uv.lock changes, the COPY layer’s hash changes, and Docker rebuilds everything after it. Both uv sync steps run from scratch. 8-15 minutes gone.
The numbers
uv.lock changed 118 times in 30 days. That’s roughly 4 times per day. Each bust costs 8-15 minutes of CI time.
Worst case: an hour of build time per day, reinstalling packages we already had, because a team we’ve never talked to bumped requests from 2.31 to 2.32.
It’s a fire alarm that goes off every time anyone in the building cooks. You still have to evacuate.
The cache mount helps — sometimes
There’s a --mount=type=cache,target=/root/.cache/uv on the RUN line. When the runner keeps its Docker state between builds, uv skips downloading packages it already has cached. That cuts the 3-5 minute dep install to under a minute.
But on ephemeral runners — or when Docker garbage-collects because the disk is 99% full — that cache is gone. Cold install every time.
Why not just hash smarter?
The real question: why does a change to uv.lock bust the cache when mai_job’s resolved dependencies didn’t actually change?
Because Docker’s cache key is the file hash. It doesn’t know which parts of uv.lock matter to mai_job. The whole file changed, so the layer rebuilds.
A smarter approach: extract mai_job’s resolved deps from uv.lock, hash just that subset, and use it as the cache key. If the subset didn’t change, skip the rebuild. This would eliminate roughly 70% of the false busts — the ones where someone else’s dependency changed but ours didn’t.
Nobody’s built that yet. But it’s the right fix.