Bazel Torch: Option B and the Five Obstacles

TL;DR: I tried to make @pypi//torch work on Linux in Bazel using uv.override_requirement + a manually fetched CPU wheel. It took five obstacles, a custom Starlark rule, and a revelation about dependency graphs to get there. Option B works — but it taught me more about why Option A might be the right call.

This is the implementation companion to What Does Success Look Like?, which laid out the four architecture options. This post is about what happens when you actually try Option B. PR #21168 has the code.

The Plan

Option B, to recap: fetch a specific CPU torch wheel via http_file, wrap it as a Bazel target, and use uv.override_requirement to make @pypi//torch resolve to it on Linux. No changes to pyproject.toml. No lockfile bloat. Surgical. Precise.

The pitch sounds great in a design doc. Here’s what actually happened.

Obstacle 1: The Bouncer at the Private API Door

First thing you need is whl_install — the Starlark rule from aspect_rules_py that knows how to take a .whl file and turn it into a proper Bazel Python target with the right PyInfo provider. It handles extracting the wheel, setting up site-packages, wiring the imports path. Essential stuff.

One problem: it lives at @aspect_rules_py//uv/private/whl_install:rule.bzl.

See that private in the path? Buildifier — Bazel’s formatting and lint tool — has opinions about loading from private paths outside their owning package. Strong opinions. “You shall not pass” opinions.

ERROR: /bazel/third_party/torch/BUILD.bazel: bzl-visibility:
  Loading from //uv/private/whl_install is not allowed from outside //uv

There’s no public API for whl_install. It’s an internal implementation detail of the uv extension. Aspect’s rules_py exposes it for its own generated code, not for you to call from your BUILD files.

Fix:

# buildifier: disable=bzl-visibility
load("@aspect_rules_py//uv/private/whl_install:rule.bzl", "whl_install")

One suppression comment. Fifteen minutes of my life I won’t get back. But fine — you’re using a private API, you acknowledge the risk, you move on.

Lesson: when there’s no public API for what you need, you’re already off the paved road. This was foreshadowing.

Obstacle 2: Alphabetical Order, Apparently, Matters

CI failed again. Not a build error. Not a test failure. A formatting check.

Buildifier enforces alphabetical ordering of attributes in MODULE.bazel. My new http_file declaration had its attributes in the order I wrote them (URL, then SHA, then name), not the order buildifier wanted (name, then SHA, then URL, or whatever its sorting algorithm decided).

ERROR: MODULE.bazel: attributes not in alphabetical order

Fix: bazel run //bazel:format. Mechanical. Two minutes.

But here’s the meta-observation: I’m two obstacles in and I haven’t gotten to anything about torch. I’ve been fighting the linter. The actual problem — “make torch work on Linux” — hasn’t started yet. This is what build system work is actually like. The ratio of “fighting infrastructure” to “solving the real problem” is roughly 3:1. Nobody tells you this in the design doc.

Obstacle 3: The Test That Tested Nothing

CI went green. Everything passed.

I almost moved on. Almost.

Then I actually looked at the test output. The smoke test — the one that imports torch and does tensor math, the entire reason this PR exists — had this line:

SKIPPED: target_compatible_with = ["@platforms//os:macos"]

The test was macOS-only. The CI runner was Linux. The test was “passing” because Bazel skipped it entirely. Green check mark. Zero assertions executed. The functional equivalent of acing an exam by not showing up.

This is worse than a failing test. A failing test tells you something is broken. A skipped test tells you nothing — but it looks like everything is fine. It’s a lie that wears the skin of truth.

The whole point of Option B is to make torch work on Linux. The one test that validates this had target_compatible_with = ["@platforms//os:macos"], meaning it literally could not run on the platform it was designed to test.

Fix: remove the macOS restriction. Let it run on Linux.

Now it runs on Linux. Now it fails on Linux. Progress.

Obstacle 4: The Deep Bug (Empty repo_name)

This is the one that took hours. The one you can’t find by reading docs. The one that only reveals itself when you run the code and trace the failure through three layers of Starlark and Rust.

After fixing the platform restriction, the test tried to actually execute. And it crashed with:

Error: No such file or directory (os error 2)
  path: "../../../..//install/lib/python3.12/site-packages"

Notice the double // in that path. That’s the fingerprint of the bug.

Here’s what’s happening under the hood. whl_install is a Starlark rule that constructs PyInfo — the Bazel provider that tells Python targets where to find their packages. Part of constructing PyInfo involves computing an imports path, which is the directory Python should add to sys.path so that import torch finds the right files.

The rule computes this path using ctx.label.repo_name — the name of the Bazel repository the rule is running in. In normal usage, whl_install runs inside an external repository (like @pypi_torch_cpu_linux//), where repo_name returns something like pypi_torch_cpu_linux. The computed path looks sensible: pypi_torch_cpu_linux/install/lib/python3.12/site-packages.

But we’re not using whl_install in an external repository. We’re using it inside the main repo via uv.override_requirement. And in Bazel 8 with bzlmod, the main repository’s repo_name is… empty. An empty string. "".

So the path becomes: ""/install/lib/python3.12/site-packages → which normalizes to /install/lib/python3.12/site-packages — an absolute path starting from the filesystem root.

Then the Rust-based venv tool in aspect_rules_py tries to resolve this path relative to the workspace with a pile of ../../../.. prefixes, producing something like ../../../..//install/lib/python3.12/site-packages. That double // isn’t a typo in the error message. It’s a literal double slash from concatenating the relative prefix with the leading / of the absolute path. The filesystem looks at this, says “I have no idea what you’re talking about,” and exits.

The fix required writing a custom rule. whl_install_override.bzl — about 100 lines of Starlark that does what whl_install does, but detects the empty repo_name case and falls back to ctx.workspace_name + "/" + ctx.label.package:

if ctx.label.repo_name:
    imports_prefix = ctx.label.repo_name
else:
    # Main repo in bzlmod: repo_name is empty.
    # Fall back to workspace_name/package.
    imports_prefix = ctx.workspace_name + "/" + ctx.label.package

The result: _main/bazel/third_party/torch/install/lib/python3.12/site-packages. A path that actually exists. A path the Rust tool can resolve.

This is the kind of bug that makes you question your career choices. Not because it’s hard — the fix is six lines. But because finding it required understanding the interaction between Bazel 8’s bzlmod repository naming, Starlark’s label API, aspect_rules_py’s Rust-based venv bootstrapper, and the Unix filesystem’s handling of double slashes. No single piece of documentation covers this. You trace it or you don’t find it.

Obstacle 5: The Dependency Graph Doesn’t Care About Your Wheel Swap

After the custom rule, torch loads. The binary extensions find their shared libraries. import torch succeeds. Tensor math works. I can almost taste the green CI check.

Then:

ModuleNotFoundError: No module named 'typing_extensions'

Torch has transitive dependencies. Seven of them, to be exact:

Here’s the thing. These seven packages are in uv.lock. They’re in @pypi//. Other packages in the monorepo pull them in. They exist. But they’re not wired as dependencies of our overridden torch target.

Why? Because uv.override_requirement replaces the wheel — the binary artifact — but not the dependency graph. The original @pypi//torch (on macOS, where it resolves from the lockfile) has its transitive dependencies computed by uv from the wheel metadata. Our override target is a raw whl_install that knows nothing about torch’s METADATA file. It’s just “here are some Python files, good luck.”

And in the lockfile, these seven deps are marker-gated. pyproject.toml says torch; sys_platform == 'darwin', so uv resolves torch’s transitive deps only for macOS. On Linux, where our override lives, the lockfile doesn’t include them as torch dependencies.

You replaced the wheel but you didn’t replace the dependency graph. The override is a body transplant without rewiring the nervous system.

The plot twist: all seven deps already exist in @pypi// because other packages in the monorepo use them. So the fix is explicit, if ugly:

py_test(
    name = "test_torch_smoke",
    srcs = ["test_torch_smoke.py"],
    deps = [
        "//bazel/third_party/torch",
        "@pypi//typing_extensions",
        "@pypi//filelock",
        "@pypi//fsspec",
        "@pypi//jinja2",
        "@pypi//networkx",
        "@pypi//setuptools",
        "@pypi//sympy",
    ],
)

Every target that uses torch needs to list all seven transitive deps. Manually. Because the dependency graph doesn’t propagate through the override boundary.

This is the leaky abstraction. The whole point of a package manager is “you depend on torch, torch’s dependencies are handled automatically.” With the override, you’ve opted out of that contract. You get the wheel. You lose the graph.

The Verdict

Option B works. The smoke test passes on Linux CI. import torch succeeds. Tensor operations produce correct results. CPU torch is alive inside Bazel’s hermetic sandbox on Linux.

But look at what it cost:

Compare to Option A (adding Linux CPU wheels to pyproject.toml):

Option A makes the lockfile fatter. Option B makes every torch consumer fatter. In a monorepo with many torch targets, Option A wins. The lockfile is one file. The consumers are many.

The Lessons

Each Fix Peels Back Another Layer

The obstacle count tells the story. Buildifier lint → buildifier format → platform restriction → deep Bazel bug → dependency graph. Each fix didn’t solve the problem — it revealed the next problem. Like an onion, except the inner layers are increasingly arcane build system internals and the tears are real.

This is the nature of working at the boundary between two systems that weren’t designed for each other. Python packaging assumes pip. Bazel assumes hermetic resolution. When you try to bridge them with an override, you’re walking through every assumption both systems made, discovering the ones that don’t hold.

Green CI Is Not the Same as Working CI

The skipped-test moment was the scariest obstacle. Not because it was hard to fix — it was trivial. But because it was invisible. CI was green. The PR could have been merged. Someone would have discovered, weeks later, that torch still didn’t work on Linux, and the investigation would have started from “but it passed CI?”

A test that can’t run on its target platform is worse than no test at all. No test is honest — it says “we don’t know.” A skipped test is dishonest — it says “we checked” when it didn’t.

You Can’t Fix a Dependency Graph Problem with a Wheel Swap

…except you partially can, if you’re lucky. The seven transitive deps happened to exist in @pypi// from other packages. If they hadn’t — if torch’s transitive deps were unique to torch — we’d have needed to fetch those wheels too, write more whl_install rules, manually resolve their transitive deps… turtles all the way down.

The override swaps the artifact. It doesn’t swap the metadata. The dependency graph is computed from the lockfile, not from the wheel you hand it. This is a fundamental limitation: uv.override_requirement was designed for “use this local build of package X” in external repos where the deps are already resolved. Using it to inject an entirely new platform’s worth of dependencies is pushing it past its design intent.

The Honest Trade-Off

Option B is an engineering achievement and a maintenance liability. It proves the mechanism works. It also proves the mechanism has sharp edges — private APIs, manual dep management, custom rules for upstream bugs.

For a spike or a proof of concept? Perfect. For long-term production use in a monorepo with dozens of torch consumers? Option A is architecturally cleaner. Accept the lockfile bloat. Let the resolver do its job. Don’t fight the dependency graph — fix it at the source.

What’s Next

This PR (#21168) stays as a proof of concept. The actual path forward is likely a combination:

1. Option A — remove the sys_platform == 'darwin' gate, let Linux CPU wheels into the lockfile, get a correct dependency graph
2. Path A from Part 3 — lazy torch imports in mai_config to unblock the 50 innocent bystanders
3. Keep the custom rule in back pocket — if we ever need fine-grained control over which torch variant Bazel uses (CPU vs CUDA vs ROCm), the whl_install_override.bzl pattern is there

The build graph will get correct. The tests will run in the right place. We just had to try the hard way first to appreciate why the easy way is actually the right way.

Previous in series: What Does Success Look Like? | The Resolution Pipeline