Bazel Torch: Three Ways to Smuggle torch In

TL;DR: I ran three experiments to get torch working in a Bazel-managed Python monorepo. Option A (standard PyPI wheels) is clean but broken on Linux. Option B (custom Starlark override) is fully hermetic but 25 files changed. Option C (Docker system torch) breaks hermeticity but matches how we already work. All three got CI green. Recommendation: C now, A long-term, B as backup.

This is the solutions companion to 62% of Our Packages Are Blocked by torch. That post mapped the wall. This one is about the three different-shaped holes I tried to punch through it. See also Selling a Technical Problem to Your Team for the communication strategy around presenting these results.

The Premise

62% of our 95-package monorepo is blocked by GPU/native dependencies. The single biggest bottleneck is torch — 27 packages depend on it directly, and a config package called mai_config cascades the blockage to ~50 packages transitively.

I set up three separate repo clones. Three approaches. Three PRs. The goal: get Bazel to resolve torch, run tests, CI green. Then compare tradeoffs.

All three succeeded. The tradeoffs are what make this interesting.

Option A: The Clean Way (That Doesn’t Fully Work)

PR #21167 | ~5 files changed

The simplest possible approach: add Linux CPU torch wheels to uv.lock through standard PyPI resolution. No custom Bazel rules. No special infrastructure. Just… torch as a normal package.

How It Works

In the root pyproject.toml, change the torch dependency from macOS-only to cross-platform:

# Before
"torch >= 2.7.0; sys_platform == 'darwin'"

# After
"torch >= 2.7.0"

Let uv lock resolve it. Let rules_py pick it up from uv.lock. Done.

What Happened

macOS: CI green. Torch resolves as a normal PyPI wheel. Tests pass. Beautiful.

Linux: broken. The resolution produces a lock file where torch’s platform markers — manylinux2014_x86_64, linux_x86_64, CUDA variant selectors — clash with how rules_py interprets markers internally. UV resolves markers one way, rules_py interprets them another way. They’re not individually wrong. They just disagree.

I had to revert the Linux change.

The Gap

This is a marker-combining problem. UV’s lock file expresses platform constraints as rich boolean expressions over environment markers. rules_py flattens these into simpler per-platform selections. When a package like torch has dozens of variant wheels with overlapping markers (CPU vs CUDA, manylinux version, Python version), the flattening loses information.

There’s discussion in the rules_py issue tracker. No fix on the roadmap.

Verdict

Partially works. macOS only. This is the right long-term answer — torch as a normal dependency, resolved by standard tooling, no special cases. But “long-term” means “when rules_py catches up.” Could be months. Could be longer.

Option B: The Hermetic Way (That Requires Custom Starlark)

PR #21168 | 25 files changed

If Bazel can’t figure out torch on its own, hand-deliver it. Fetch the CPU wheel directly from download.pytorch.org via http_archive, then wire it into Bazel’s module system with a custom Starlark rule.

How It Works

Three layers:

Layer 1: Fetch the wheel

# MODULE.bazel
http_archive(
    name = "torch_cpu_wheel",
    urls = ["https://download.pytorch.org/whl/cpu/torch-2.7.0-cp312-cp312-manylinux_2_17_x86_64.whl"],
    sha256 = "...",
    type = "zip",  # whl files are zips
)

Layer 2: Tell Bazel how to use it

A custom rule — whl_install_override.bzl — that takes the extracted wheel contents and creates a proper py_library target. This rule understands torch’s internal layout: the .so files in torch/lib/, the Python modules in torch/, the metadata in torch-2.7.0.dist-info/.

whl_install_override(
    name = "torch",
    wheel = "@torch_cpu_wheel",
    deps = ["@pypi//numpy", "@pypi//typing_extensions"],
)

Layer 3: Override the pip resolution

# Tell rules_py: when someone depends on @pypi//torch, use our override instead
uv.override_requirement(
    requirement = "torch",
    target = "@torch_cpu_override//:torch",
)

What Happened

CI green. Both macOS and Linux. Full dependency resolution. The entire 59-package set can see torch in the Bazel graph.

But.

25 files changed. A custom Starlark rule that understands torch’s internal wheel structure. A workaround for a Bazel 8 bzlmod bug where module extensions in overrides don’t propagate correctly. And a maintenance contract: every time torch releases a new version, someone updates the URL, the SHA, and possibly the .so file list.

The Maintenance Cost

This is the part that matters more than the cleverness.

Each GPU package needs its own override. Torch is one. Flash-attn is another. Triton is another. The pattern doesn’t scale linearly — it scales manually.

Verdict

Fully works. Fully hermetic. Bazel controls the entire dependency graph, no external state. But it’s custom infrastructure. You’re writing a package installer inside Bazel. The kind of thing that works great until the person who wrote it changes teams.

Option C: The Pragmatic Way (That Purists Will Hate)

PR #21169

Instead of making Bazel understand torch, make Bazel aware that torch is already installed. Don’t smuggle torch into the build graph — leave it outside and open a side door.

How It Works

Two custom rules:

local_runtime_repo — a repository rule that inspects the host system’s Python environment at analysis time. It looks for torch (and other system-installed packages) and creates stub py_library targets that point to the existing installation.

local_runtime_repo(
    name = "system_torch",
    packages = ["torch", "flash_attn", "triton"],
    python = "/usr/bin/python3",
)

py_venv_test — a test rule that creates a virtual environment with include_system_site_packages=True. This is the key mechanic: Python’s venv inherits system packages. Tests run inside this venv, so they can import torch from the Docker image’s system installation.

py_venv_test(
    name = "test_model_forward",
    srcs = ["test_model_forward.py"],
    deps = [":model_lib", "@pypi//numpy"],
    system_deps = ["@system_torch//:torch"],
    include_system_site_packages = True,
)

What Happened

CI green. Tests import torch from the Docker container’s pre-installed packages. Bazel manages the Python code and pure-Python dependencies. Docker manages the GPU libraries.

This is, philosophically, an admission. Bazel doesn’t control torch. Docker does. The build graph has a hole in it — a dependency that Bazel can see but doesn’t manage.

Why This Isn’t as Bad as It Sounds

The knee-jerk reaction is “you broke hermeticity.” And yes, technically, I did. But let’s be honest about what hermeticity we actually had.

Our GPU tests already run this way. The Docker images already have torch, flash-attn, and triton pre-installed. The current CI workflow is:

1. Build Docker image with pip-installed GPU packages
2. Run pytest inside that container
3. Hope the pip versions match what pyproject.toml says

Option C doesn’t break something we had. It formalizes something we already do. The contract shifts from implicit (“the Docker image probably has the right torch”) to explicit (“Bazel checks that torch exists at analysis time, and the test rule declares the dependency”).

It’s less hermetic than pure Bazel. But it’s more hermetic than our current setup.

The Docker-Layer Contract

The reproducibility guarantee moves from the Bazel layer to the Docker layer:

Same underlying contract. Better surface area.

Verdict

Fully works. Lowest maintenance. Matches existing workflow. But breaks Bazel hermeticity — the test outcome depends on the Docker image, not the build graph. If you update the base image and torch changes, tests might break and Bazel won’t know why until runtime.

The Comparison

The Recommendation

Start with C. Plan for A. Keep B in your back pocket.

Option C unblocks 59 packages immediately with the lowest maintenance cost. It aligns with how GPU tests already work — Docker manages native packages, Bazel manages Python code. It’s honest about the boundary between what Bazel can control and what it can’t.

Option A is the right long-term answer. When rules_py adds proper handling for complex platform markers and native wheels, torch should just be another @pypi// dependency. No special cases. No overrides. But that day isn’t today.

Option B is insurance. If we ever need full hermeticity — reproducible builds that don’t depend on Docker image state — the machinery exists and is proven. But the maintenance cost is real, and for a team our size, “custom Starlark package installer” is the kind of infrastructure that sounds cool until you’re the person maintaining it at 11 PM.

The Philosophical Tension

There’s a deeper question here that I don’t think has a clean answer:

Should a build system own dependencies it can’t build?

Torch is a 2GB package with CUDA kernels, custom C++ extensions, vendor-specific GPU code, and platform-specific binary blobs. It’s not a Python package in any meaningful build-system sense. It’s a pre-compiled runtime. Like the JVM. Like the CUDA driver itself.

Bazel’s philosophy is “hermetic builds — the build system controls everything.” But “everything” was designed for a world where dependencies are source code or simple binary artifacts. A 2GB GPU framework that only works on specific hardware with specific drivers doesn’t fit that model.

Option C’s answer is: draw the boundary honestly. Bazel manages what Bazel can manage. The runtime environment manages the rest. Document the boundary. Make it explicit. Move on.

It’s not pure. But at 62% of packages blocked, purity is a luxury.

What’s Next

The immediate next steps:

1. Land Option C infrastructure — local_runtime_repo and py_venv_test rules, plus the mai_config torch dependency audit
2. Migrate the 23 pure-Python packages — these are unblocked and waiting
3. Migrate torch-dependent packages — using Option C’s system_deps mechanism
4. Track rules_py progress — when marker-combining is fixed, evaluate switching to Option A
5. Document the upgrade path — how to move from C to A when the time comes

The goal isn’t to pick one option forever. It’s to pick the option that unblocks work now while keeping the door open for the cleaner answer later.

All three experiments are in separate PRs (#21167, #21168, #21169) with full CI results. The code is there. The tradeoffs are documented. Now it’s a team decision.

Field notes from running three parallel Bazel experiments in a 95-package Python monorepo. All three approaches, PRs, and CI results are real. The sys_platform == 'never' hack from the blocker analysis is also real, and Option C is essentially its grown-up replacement.

Previously: 62% of Our Packages Are Blocked by torch — the blocker analysis. See also: Selling a Technical Problem to Your Team — how I presented this work across two meetings.