Our pretraining image takes 40 minutes to build. Someone suggested Bazel’s rules_img — a ruleset that builds OCI images without Docker. Granular layer caching, parallel assembly, lazy push to registries. Sounds like exactly what we need.
It’s not.
The wrong bottleneck
rules_img optimizes how container layers get assembled and pushed. That part of our build takes maybe 30 seconds. The other 39 minutes and 30 seconds? Compiling CUDA code from source.
Flash Attention, Triton, MMCV, DeepEP — each one does a git clone and builds from source inside the Dockerfile. They’re sequential. Each takes 5-20 minutes. Bazel can’t make nvcc go faster.
It’s like buying a faster delivery truck when the bottleneck is the factory floor.
The migration tax
Even if layer assembly were the bottleneck, switching to rules_img means rewriting the entire build in Bazel. Our Python packages are managed with uv. CUDA kernels compile with CMake and ccache. The NGC base image is a 15GB PyTorch container from NVIDIA.
To use rules_img, you’d wrap all of that as Bazel targets — or use genrules, which defeats the purpose of hermeticity. That’s weeks of work for an optimization that doesn’t address the actual slow part.
What rules_img is actually good for
To be fair, rules_img solves real problems. If your bottleneck is layer assembly, cache invalidation at the layer level, or pushing large images repeatedly — it’s excellent. The “build without the bytes” pattern with remote build execution is genuinely clever.
But you have to match the tool to the bottleneck. Our bottleneck is CUDA compilation, not image assembly. BuildKit already handles our layer caching reasonably well.
The actual fix
The 40-minute build has two real problems: sequential CUDA compilation and a lockfile that busts the cache 4 times a day. Neither involves Bazel.
More on those in the next posts.