When GPU Showed Up in Our Python Bazel Monorepo

2026/02/22

BUILDmigrationtorch

TL;DR: A Python Bazel monorepo is genuinely pleasant to maintain — until CUDA enters the picture. Then Bazel’s core promise (hermeticity, reproducibility, one-command builds) starts cracking at the seams, and you end up duct-taping it back together with Docker, host symlinks, and comments that say “horrible hack” in production code.

The Before Times

Before GPU workloads arrived, our Python Bazel monorepo was… fine. Actually better than fine. We had hermetic builds, solid caching, py_test targets that ran the same everywhere. You could bazel test //... and go get coffee. The dependency graph was pure Python — pip packages resolved through rules_python, pinned, reproducible. Life was good.

Then someone said “we need to train models.”

What Actually Changes

Here’s what your clean, well-maintained Python Bazel monorepo has to absorb once GPU/AI workloads show up. Not in theory. In practice, in production, with the comments to prove it.

1. Your Build Gets a GPU Toggle

The first thing you need is a way to say “this build targets a GPU.” Sounds simple. You end up with a tri-state flag because it’s not just “GPU or not” — it’s “which GPU vendor”:

string_flag(name = "graphic_accelerator", build_setting_default = "missing")
config_setting(name = "graphic_accelerator_cuda", flag_values = {":graphic_accelerator": "cuda"})
config_setting(name = "graphic_accelerator_rocm", flag_values = {":graphic_accelerator": "rocm"})

Now every cc_library that touches anything GPU-adjacent uses select() to pick dependencies and compiler defines based on this flag. One flag gates the entire GPU dependency tree. It’s elegant, honestly — until you realize this flag now propagates through hundreds of targets and you’re debugging why some transitive dep four layers deep didn’t get the right defines.

2. Hermeticity Goes Out the Window

This is the big one. Bazel’s entire value proposition is hermetic builds — everything declared, everything reproducible, nothing leaking in from the host. GPU blows that up.

You end up writing repository_rules that symlink directly from the host filesystem:

repository_ctx.symlink("/usr/local/cuda", "cuda")  # non-hermetic!

That’s CUDA. For PyTorch, it gets worse:

# "A horrible hack to use Torch from the local install.
#  About as non-hermetic as it gets. Oh well."
torch_result = repository_ctx.execute(
    [python, "-c", "import torch; print(torch.__file__)"]
)
repository_ctx.symlink(torch_dir, "torch")

That comment is real. It’s in production. And it’s accurate — we’re running Python inside a repository rule to discover where PyTorch installed itself, then symlinking it into Bazel’s sandbox. This is the Bazel equivalent of reaching through a clean room wall to grab a tool off the workbench outside.

Why? Because CUDA toolkit is 5-8 GB. PyTorch wheels with CUDA support are 2+ GB. You can’t reasonably download and extract these as hermetic Bazel dependencies on every build. The bandwidth, the storage, the cache invalidation — it doesn’t work. So you accept the host dependency and move on.

3. Docker Becomes Your Real Hermetic Boundary

Since Bazel can’t be hermetic about GPU deps, something else has to be. That something is Docker.

Your CI worker image becomes the actual reproducibility boundary:

# Base differs by architecture — yes, different CUDA versions
# amd64: nvidia/cuda:12.9.1-cudnn-devel-ubuntu24.04
# arm64: nvidia/cuda:13.0.0-cudnn-devel-ubuntu24.04

Inside that image you have:

The Dockerfile is now your WORKSPACE file. It’s where the real dependency pinning happens. Bazel just trusts that whatever’s in the container is correct. You’ve moved the hermeticity boundary from Bazel (build-level) to Docker (environment-level).

This works. It’s just… not what the Bazel brochure promised.

4. You’re Writing C++ Now

Pure Python monorepo? Not anymore.

AI workloads need native extensions — custom CUDA kernels, optimized attention implementations, quantization routines. Suddenly you’re writing (or wrapping) C++ code that links against CUDA runtime:

cc_shared_library(
    name = "my_cuda_extension",
    deps = [":cuda_kernels"],
    # ... platform-specific linking
)

Your build rules now handle:

That last one is fun. Your Python monorepo now has opinions about network fabric.

5. Architecture-Specific Everything

In pure Python land, architecture barely matters. py_library doesn’t care if it’s amd64 or arm64.

With GPU, architecture is everywhere:

That last point means your build system implicitly encodes a hardware support matrix. Your BUILD files know what GPUs exist in your fleet.

6. CI Gets Expensive and GPU-Aware

Before GPU, CI was straightforward — spin up containers, run tests, done. Now:

agents = AgentTargeting(APPLIED_BUILDKITE_BAZEL_RUNNER_CUDA_OPT_QUEUE),
extra_bazel_options = [
    "--graphic_accelerator=cuda",
    "--nb_extension_mode=hybrid",
]

You need dedicated GPU worker queues. GPU instances cost 5-10x what CPU instances cost. You can’t just “run all tests on every PR” anymore — GPU test time is expensive, so you start building test selection, GPU test tagging, and scheduling heuristics.

Your CI bill goes up. Dramatically.

7. Two Paths for Everything

This is the pattern that surprised me most. For PyTorch alone we ended up with two build modes:

Both exist because neither works in all cases. The hybrid path needs the right PyTorch pre-installed (Docker’s job). The prebuilt path avoids that dependency but requires maintaining artifact storage. You toggle between them with another flag (--nb_extension_mode=hybrid vs prebuilt).

This “two paths” pattern repeats everywhere once GPU enters the picture. Every GPU dependency that Bazel can’t manage hermetically gets this treatment: one path for “it’s already on the host,” another for “fetch it from somewhere.”

The Meta-Insight

Here’s what I think is actually going on.

Bazel was designed in a world where all dependencies are software: source code, libraries, tools. Software can be downloaded, checksummed, cached, and sandboxed. Bazel is brilliant at this.

GPU introduces hardware as a dependency. The CUDA toolkit isn’t just software — it’s a contract between the driver on the host, the runtime in the container, and the physical GPU on the machine. You can’t sandbox hardware. You can’t checksum a GPU. You can’t download an A100.

So what do you do? You build a different kind of hermetic boundary. Instead of Bazel sandboxing each build action, Docker sandboxes the entire environment. Instead of Bazel fetching dependencies, you pre-bake them into images. Instead of one clean dependency graph, you get a layered system:

Layer 3: Bazel (manages Python deps, builds, tests)
Layer 2: Docker (manages CUDA, PyTorch, native libraries)
Layer 1: Host (manages GPU driver, hardware)

Each layer trusts the one below it. Bazel trusts Docker to provide the right CUDA. Docker trusts the host to provide the right driver. It works — it’s just a more complex system than what you started with.

Is It Worth It?

Yes. Even with all the hacks, Bazel still gives you:

The alternative — separate build systems for CPU and GPU code, cmake for native extensions, shell scripts for Docker, something else for CI — is worse. I’ve seen it. It’s worse.

But you should go in with eyes open. Your clean Python monorepo will absorb a lot of complexity. The WORKSPACE file gets longer. The Docker images get heavier. The CI gets more expensive. And somewhere in your codebase, there will be a comment that says “horrible hack” — and it’ll be telling the truth.

Written from the experience of maintaining a Python+GPU Bazel monorepo. The code samples and comments are from a real production codebase. The “horrible hack” comment is verbatim.