Why Your Bazel CI Shouldn’t Share a Docker Image with Pytest

We had a fat Docker image. It worked great for pytest. When we started adding Bazel CI, the obvious move was to reuse it. Same monorepo, same tests, same CUDA stack. Why build a second image?

It took us about three weeks to figure out why. The short version: when two CI systems manage dependencies in fundamentally different ways, sharing an image is not simplicity. It’s a correctness trap with a slow build time penalty on top.

The fat image and the thin image

Our monorepo runs GPU tests. Both CI systems need CUDA, cuDNN, NCCL, PyTorch, Flash Attention 3, and Triton. All of that lives in a shared base layer built from NVIDIA’s NGC container. That layer doesn’t change often and takes ~30 minutes to build cold. Neither CI system argues about it.

The divergence starts one layer up.

The fat image does a lot of work. It runs a multi-phase uv sync that installs every Python dependency into a venv, then precompiles CUDA kernels so pytest doesn’t have to build them at test time. The thin image installs ccache and a Python symlink. That’s it. 21 lines of Dockerfile.

The phantom dependency problem

This is the one that actually matters. The build time stuff is annoying but manageable. The correctness issue is not.

Here’s how Bazel handles Python deps: it reads uv.lock, fetches the exact wheels it needs at build time, and injects them into a per-target virtual environment. If a test imports something that isn’t declared in its deps, Bazel should fail with an import error. That’s the whole point of hermetic builds. Undeclared dependencies are bugs, and Bazel is supposed to catch them.

Now here’s what happens when Bazel runs inside the fat image.

The fat image has a /venv with ~1800 packages installed via uv sync. The venv was created with --system-site-packages so it can see NGC’s packages (torch, etc.). Bazel’s test targets use include_system_site_packages=True to pick up torch and CUDA bindings from the system. But that flag doesn’t distinguish between “NGC system packages” and “everything in /venv.” It exposes the entire venv.

So a test can import pandas without declaring @pypi//pandas in its BUILD file. It works, because pandas happens to be installed in /venv. The test passes. CI is green. Nobody notices the missing dependency.

Then one of three things happens:

1. The image gets rebuilt and /venv changes slightly. The test breaks. Nobody understands why.
2. A developer runs the test on their machine, which doesn’t have the venv. The test breaks. They blame Bazel.
3. You finally migrate away from the fat image. Dozens of tests break simultaneously because their BUILD files were lying the whole time.

This is the opposite of hermeticity. The fat image turns Bazel’s dependency enforcement into a suggestion.

Cache invalidation: you’re hashing the wrong thing

The fat image precompiles CUDA kernels. This is genuinely useful for pytest, because it means test runners don’t spend 10 minutes on nvcc before running a single test. But it means the image hash has to include CUDA kernel source files:

hashFiles('docker/Dockerfile',
          'docker/pretraining/Dockerfile',
          'mai_kernels/csrc/**',
          'mai_kernels/src/**',
          'mai_kernels/CMakeLists.txt',
          'mai_kernels/pyproject.toml')

Any change to a .cu file invalidates the image cache and triggers a full rebuild. For pytest, this makes sense. The precompiled .so files are stale, so you need a new image.

Bazel doesn’t use those .so files. Bazel compiles CUDA kernels itself at build time, with ccache on NFS for cross-run caching. The precompiled artifacts in the fat image are dead weight. But because Bazel was using the fat image, every CUDA kernel change triggered a ~10 minute Docker rebuild that Bazel never needed.

The thin image’s hash is just:

hashFiles('docker/Dockerfile',
          'docker/bazel-gpu/Dockerfile',
          'docker-bake.hcl')

No kernel sources. No lock files. The only thing that triggers a rebuild is changing the Dockerfile itself or the NGC base. In practice, that’s maybe once or twice a month instead of several times a week.

What other Bazel migrations found

We’re not the first team to discover this. Uber documented their Bazel migration (2019-2020) and explicitly recommended against sharing images between the legacy system and Bazel. Their reasoning was the same: fat images masked dependency declaration bugs in BUILD files. Tests would pass in CI but fail on developer machines.

Stripe’s Bazel migration hit the same wall. When the Bazel image was too fat, developers accidentally relied on implicit dependencies that weren’t declared. Green CI, broken local builds.

The pattern that emerged across these migrations, and in public Bazel CI setups (rules_python, Envoy, gRPC), is straightforward: the image provides what Bazel can’t manage, and nothing else. For most projects, that’s the OS, the compiler toolchain, and Bazelisk. For GPU workloads, add the CUDA stack and GPU drivers. Python packages? Bazel fetches those. CUDA kernel compilation? Bazel does that. The image is an environment container, not a dependency container.

The decision framework

So when should parallel CI systems share an image?

Share when they discover packages the same way. If both systems use the same venv and both expect packages to be pre-installed, a shared image eliminates the “two environments” class of bugs with no downside.

Don’t share when they have different dep management strategies. If one system uses a pre-built venv and the other uses a lockfile-based resolver that fetches its own deps, sharing means one system carries dead weight while the other gets phantom dependencies that undermine its correctness guarantees.

The litmus test is simple: does the second system use the extra stuff in the fat image? If yes, share. If the extra stuff is invisible to the second system (it fetches its own deps), or worse, actively harmful (it leaks undeclared deps), don’t share.

It’s like asking whether your production app server should double as your build machine. You could do it. It would technically work. But the environments have different requirements, and conflating them creates maintenance burden and subtle bugs that only surface when you can least afford to debug them.

What we actually did

We added a 21-line Dockerfile. It inherits from the same NGC-based shared layer (so torch, CUDA, Flash Attention 3, and Triton are all identical), installs ccache, adds a Python symlink that Bazel’s toolchain registration expects, and does nothing else.

FROM shared

RUN apt-get update && apt-get install -y ccache && rm -rf /var/lib/apt/lists/*
RUN ln -sf /usr/bin/python3 /usr/local/bin/python3

That’s the interesting part. Everything else is plumbing.

Build time went from ~10 minutes to ~1 minute. Rebuild frequency went from several times a week (triggered by CUDA kernel changes) to roughly monthly (triggered by NGC or Dockerfile updates). And we stopped needing a runtime hack that symlinked the venv Python into Bazel’s expected path, which was a small correctness bomb waiting to go off.

The thin image also killed the phantom dependency problem. With no /venv in the image, include_system_site_packages=True only exposes what NGC provides: torch, triton, and the CUDA bindings. If a test imports something that isn’t declared in its BUILD file and isn’t part of the GPU stack, it fails immediately. Which is exactly what you want during a Bazel migration, because every silent pass on an undeclared dep is a bug you’ll find later in worse conditions.

Twenty-one lines. Faster builds. Fewer phantom deps. Correct cache invalidation. Sometimes the right abstraction is just removing the stuff that shouldn’t have been there.