Bazel Torch: Option A, the Simplest Option That Wasn’t

TL;DR: “Just remove the darwin gate and let uv resolve torch for all platforms.” That’s the pitch. Seven phases, a false victory, and one humbling CI crash later, it works — but the path from “simplest option” to “working option” went through a CPU index rabbit hole, a uv sync blast radius discovery, a “lazy CUDA” myth that only held on macOS, and a final fix involving lockfile post-processing and an aspect_rules_py patch. Option A isn’t one option. It’s a family of sub-approaches, and most of them are wrong.

This is the implementation companion to What Does Success Look Like?, which laid out the four architecture options and five levels of “done.” Parallel to the Option B and Option C posts. PR #21167 has the code.

The Plan

Option A is the one that sounds like it shouldn’t need a blog post.

Remove the sys_platform == 'darwin' gate from pyproject.toml. Run uv lock. Let aspect_rules_py pick up the newly-resolved Linux manylinux wheels. @pypi//torch now has a Linux arm in its select_chain. Smoke test runs everywhere. Done. Three minutes. Maybe five if you read the diff.

Here’s what actually happened.

Phase 1: Just Remove the Gate

The darwin gate in pyproject.toml looked like this:

override-dependencies = [
    "torch; sys_platform == 'darwin'",
    "torchvision; sys_platform == 'darwin'",
    "torchaudio; sys_platform == 'darwin'",
]

Translation: “uv, only resolve torch for macOS. On Linux, pretend it doesn’t exist.” This was the original workaround — production Linux uses system-installed torch inside Docker, so the lockfile doesn’t need it. Except now Bazel needs it, because @pypi//torch has to resolve on all platforms.

Remove the gate. Pin the version. Run uv lock:

override-dependencies = [
    "torch==2.7.0",
    "torchvision==0.22.0",
    "torchaudio==2.7.0",
]

uv lock succeeds. The lockfile now includes manylinux_2_28_x86_64 and manylinux_2_28_aarch64 entries. The generated select_chain covers all platforms. bazel build @pypi//torch passes on macOS. bazel test //bazel/tests:test_torch_smoke passes on macOS.

This took ten minutes. I briefly considered closing the PR.

Then I looked at what uv actually resolved for Linux.

Standard PyPI torch for Linux is CUDA-bundled. 865MB. Not CPU-only. PyPI doesn’t host CPU-only Linux wheels — those live on PyTorch’s own index (download.pytorch.org/whl/cpu). When you pip install torch on Linux from PyPI, you get the CUDA variant by default.

865MB of CUDA runtime that a CPU-only CI runner can’t use.

And that CUDA torch drags in 14 nvidia-* transitive dependencies — nvidia-cudnn-cu12, nvidia-cublas-cu12, nvidia-cuda-runtime-cu12, and friends. These packages expect CUDA libraries to exist on the host. On a GitHub-hosted runner with no GPU, import torch crashes with:

OSError: libcudart.so.12: cannot open shared object file: No such file or directory

First fix: block all 14 nvidia-* packages with sys_platform == 'never' overrides:

"nvidia-cuda-nvrtc-cu12; sys_platform == 'never'",
"nvidia-cuda-runtime-cu12; sys_platform == 'never'",
"nvidia-cuda-cupti-cu12; sys_platform == 'never'",
"nvidia-cudnn-cu12; sys_platform == 'never'",
"nvidia-cublas-cu12; sys_platform == 'never'",
# ... 9 more

The sys_platform == 'never' trick: uv evaluates this marker, finds no platform matches, and excludes the package from resolution. It’s a hack dressed up as a PEP 508 marker. But it works, and it’s the only way to tell uv “resolve this package but not its CUDA transitive deps.”

At this point I had a lockfile with Linux torch but the smoke test was still gated to macOS (from a previous PR). Green CI, zero assertions executed on Linux. The test was passing by not running — the exact trap I wrote about in the Option B post.

Phase 2: The CPU Index Rabbit Hole

The CUDA-bundled wheel was too heavy and too dependent on system libraries. The obvious fix: use PyTorch’s CPU index to get CPU-only wheels (~300MB, no nvidia-* deps, no CUDA requirement).

uv lock --index "https://download.pytorch.org/whl/cpu" \
  --index-strategy unsafe-best-match --upgrade-package torch

This worked! Sort of. The lockfile now had torch==2.7.1+cpu for Linux and torch==2.7.0 for macOS. CPU-only, small, clean. The nvidia-* overrides could be removed entirely.

But that +cpu suffix. That innocent little +cpu local version suffix. It was about to ruin the next three days.

Problem 1: Percent-Encoding

Wheel filenames encode the + as %2B per URL encoding rules. So the wheel is named torch-2.7.1%2Bcpu-cp312-cp312-manylinux_2_17_x86_64.whl. The Rust unpack binary in aspect_rules_py received this filename and tried to parse 2.7.1%2Bcpu as a PEP 440 version. It’s not. It’s a URL-encoded version string. The parser choked.

Fix: write a patch for aspect_rules_py’s extension.bzl to URL-decode %2B back to + in wheel filenames before parsing.

Problem 2: Multi-Version Merge

The lockfile now had two entries for torch — 2.7.0 (macOS, from PyPI) and 2.7.1+cpu (Linux, from CPU index). aspect_rules_py generates one Bazel repository per package. Two versions meant the merge logic kicked in — combining both into a single whl_install repo with a cross-platform select.

The merge used name-only dedup for dependencies. Both torch versions depend on typing-extensions. macOS torch says typing-extensions; sys_platform == 'darwin'. Linux torch says typing-extensions; sys_platform == 'linux'. The merge saw “same package name” and kept only the first entry. typing-extensions ended up macOS-only.

On Linux CI: ModuleNotFoundError: No module named 'typing_extensions'.

Fix: OR-combine markers when the same dependency appears in both versions.

Problem 3: Nested Parentheses

The OR-combine created markers like (sys_platform == 'darwin') or (sys_platform == 'linux' and python_version >= '3.12'). Looks fine. Except aspect_rules_py’s PEP 508 marker parser has a bug with nested parentheses. The naive wrapping — (A) or (B) — created double-nesting when B itself contained parens from torch’s original markers.

Fix: only wrap marker operands in parens when they have a top-level and. Flat or chains stay flat.

Problem 4: Stale Hunk Headers

After all the fixes to the patch, the hunk line numbers were wrong. The patch tool matched on line offsets, and the added helper functions shifted everything. CI failed with CONTENT_DOES_NOT_MATCH_TARGET.

Fix: regenerate the entire patch from a clean diff against upstream v1.8.4.

Four bugs from a +cpu suffix. A URL encoding issue, a merge logic bug, a parser bug, and a stale patch. Each one required a different kind of knowledge — URL standards, PEP 508 marker algebra, Starlark string manipulation, and git diff mechanics. All because CPU-only torch puts a local version suffix in its wheel filename.

This is the fractal nature of build system problems. You think you’re solving “get CPU torch into Bazel.” You’re actually solving “make a Rust binary understand percent-encoded PEP 440 versions in filenames that get merged across platform variants with marker expressions that need to be parenthesized but not double-parenthesized.”

Phase 3: Retreat

After four patches to aspect_rules_py, I stepped back and looked at what I’d built. A dual-version lockfile (2.7.0 + 2.7.1+cpu) that required custom patches for URL decoding, multi-version merging, marker combining, and parenthesization. Every uv lock would regenerate this complexity. Every rules_py version bump could break the patches.

The maintenance cost was already higher than Option B’s custom Starlark rule. The whole point of Option A was supposed to be “no custom infrastructure.” I had more custom infrastructure than any other option.

Decision: revert to standard PyPI torch. Single version (2.7.0), single lockfile entry, no patches. The CUDA-bundled wheel is 865MB but the lockfile is clean.

# Simplified: standard PyPI torch, nvidia-* blocked
"torch==2.7.0",
"torchvision==0.22.0",
"torchaudio==2.7.0",
"nvidia-cuda-nvrtc-cu12; sys_platform == 'never'",
# ... 13 more nvidia-* overrides

But the smoke test went back to macOS-only. Standard PyPI torch on Linux still needed nvidia-* at import time… or did it?

Phase 4: The uv sync Surprise

Before tackling the import question, I discovered a bigger problem.

Changing pyproject.toml’s override-dependencies doesn’t just affect Bazel. It affects uv sync. Every developer. Every CI runner. Every Docker-based test job.

When you remove the darwin gate, uv sync on a Linux CPU test runner installs CUDA-bundled torch from standard PyPI. This overwrites the pre-installed CPU-only torch that the Docker image ships with. CUDA torch needs nvidia-* runtime libs. CPU test runners don’t have them. Tests crash.

The blast radius of touching pyproject.toml in a monorepo is not limited to Bazel. It’s everyone. uv sync is the first thing every CI job runs. Change what it resolves and you’ve changed every CI pipeline simultaneously.

I had to revert pyproject.toml and uv.lock back to match main. The darwin gate went back on. All progress undone.

Lesson: in a monorepo, pyproject.toml is a shared resource with global scope. You can’t “just” change a dependency constraint because “just” means “for 200 CI jobs across every team in the org.” The change needs to be compatible with every consumer of uv sync, not just Bazel.

Phase 5: tool.uv.sources (Almost Right)

Back at the drawing board. New approach: use [tool.uv.sources] with platform markers to route Linux resolution to the PyTorch CPU index while keeping macOS on PyPI.

[[tool.uv.index]]
name = "pytorch-cpu"
url = "https://download.pytorch.org/whl/cpu"
explicit = true

[tool.uv.sources]
torch = [
    { index = "pytorch-cpu", marker = "sys_platform == 'linux'" },
]
torchvision = [
    { index = "pytorch-cpu", marker = "sys_platform == 'linux'" },
]
torchaudio = [
    { index = "pytorch-cpu", marker = "sys_platform == 'linux'" },
]

This is elegant. Linux gets CPU-only wheels from the CPU index. macOS gets standard wheels from PyPI. uv sync on Linux installs CPU torch. uv sync on macOS installs macOS torch. Bazel gets both platform arms. Nobody’s existing workflow breaks.

Except… the CPU index again means torch==2.7.1+cpu for Linux. Same +cpu suffix. Same percent-encoding. Same multi-version merge issues from Phase 2.

Same rabbit hole, different entrance.

Phase 6: The Lazy CUDA Revelation (That Wasn’t)

At this point I’d been through the CPU index twice and backed out twice. The fundamental tension: CPU-only wheels are small and clean but create dual-version lockfiles that break tooling. PyPI wheels are single-version but bundle CUDA and need nvidia-* libs at import time.

Except… do they?

I tested it. On my macOS machine (no CUDA, no nvidia anything), with the standard PyPI torch:

>>> import torch
>>> torch.__version__
'2.7.0'
>>> torch.tensor([1.0, 2.0, 3.0]).sum().item()
6.0

It worked. No crash. No libcudart.so.12 error.

I wrote up the theory: CUDA libraries are lazy-loaded. Torch doesn’t load CUDA at import time. It loads CUDA when you call .cuda(), or torch.cuda.is_available(), or anything that actually touches the GPU. For CPU-only tensor math — torch.tensor(), torch.zeros(), matrix operations — the CUDA libraries are never loaded. Block the nvidia-* packages with sys_platform == 'never', and the CUDA torch wheel works everywhere. Clean. Elegant. Final.

I pushed the PR with confidence.

Then CI ran on Linux x86_64.

OSError: libcudart.so.12: cannot open shared object file: No such file or directory
ValueError: libcublas.so.*[0-9] not found in the system path

The “lazy CUDA” theory was wrong. Or more precisely — it was right on macOS for a reason that had nothing to do with laziness.

Here’s what actually happens. The standard PyPI torch wheel for macOS is already CPU-only. There is no CUDA on macOS. Apple dropped CUDA support years ago. The macOS wheel ships with no CUDA shared libraries, no _load_global_deps() call, no _preload_cuda_deps(). When I tested import torch on macOS and it worked, I was testing a CPU-only wheel and declaring CUDA “lazy-loaded.”

The standard PyPI torch wheel for Linux is CUDA-bundled. On import, it calls torch._C → _load_global_deps() → _preload_cuda_deps(), which does ctypes.CDLL("libcudart.so.12"). This is not lazy. This is eager. This happens before your first line of Python after import torch.

I tested on the wrong platform and drew a universal conclusion from a platform-specific behavior. The macOS test was literally testing nothing about the Linux question. Classic “works on my machine” with extra steps.

Phase 7: The Real Fix

So the CUDA-bundled PyPI wheel doesn’t work on CPU-only Linux runners. The CPU index creates dual-version lockfile nightmares. And you can’t change pyproject.toml without nuking uv sync for the whole org.

The actual solution came from a different angle: don’t fight the lockfile format. Post-process it.

Part 1: Swap Linux Wheel URLs in uv.lock

The idea: keep a SINGLE [[package]] entry for torch at version 2.7.0. Run uv lock normally (PyPI resolves CUDA-bundled wheels for Linux). Then post-process the lockfile to replace the Linux wheel URLs — swap the 865MB CUDA-bundled wheel URLs from PyPI with the ~176MB CPU-only equivalents from download.pytorch.org/whl/cpu.

macOS and Windows wheels stay untouched (macOS is already CPU-only, Windows kept for completeness).

Why not use [tool.uv.sources] to route to the CPU index directly? Because that creates TWO [[package]] entries — one torch==2.7.0 from PyPI, one torch==2.7.0+cpu from the CPU index. And aspect_rules_py’s _group_repos function (line 712 of extension.bzl) does this:

packages = {package["name"]: package for package in lock.get("package", [])}

Dict comprehension. Last writer wins. Two [[package]] entries for “torch” → one gets silently dropped. You end up with either macOS-only or Linux-only wheels, depending on which entry comes second in the lockfile. We MUST have a single entry.

The post-processing approach gives us that single entry. Same version string (2.7.0), same package name, just different wheel URLs for Linux. No dual-version. No +cpu suffix. No merge logic. No percent-encoding issues.

A maintenance script at bazel/scripts/patch_torch_cpu_wheels.py handles the swap. Run it after every uv lock regeneration.

Part 2: Patch aspect_rules_py for URL-Decoded Filenames

The CPU-only wheel filename is torch-2.7.0+cpu-cp312-cp312-manylinux_2_28_x86_64.whl. In the URL, + is encoded as %2B. aspect_rules_py extracts filenames via url.split("/")[-1] without URL-decoding. The Rust unpack tool then tries to parse 2.7.0%2Bcpu as a PEP 440 version and fails.

This is the same percent-encoding bug from Phase 2, but now it’s the only patch we need instead of one of four. A _url_decode_filename() helper that decodes %2B → + at all 4 locations where url.split("/")[-1] is used in extension.bzl. Small, focused, unlikely to break on version bumps.

Part 3: CI Cache Whack-a-Mole

The Bazel CI workflow has a 3-level restore-key cascade:

restore-keys: |
  bazel-${{ runner.os }}-${{ hashFiles('.bazelversion') }}-${{ hashFiles('MODULE.bazel') }}-
  bazel-${{ runner.os }}-${{ hashFiles('.bazelversion') }}-
  bazel-${{ runner.os }}-

The broadest fallback (bazel-Linux-) matched stale caches from previous runs that still had the old CUDA-only select_chain. Bazel restored the cached analysis, saw the cached torch repo, and never re-evaluated the new CPU-only wheel targets. Tests ran against phantom targets that pointed at wheels that no longer existed in the new configuration.

Fix: gh cache delete to nuke the stale caches. Fresh run. Clean resolution.

Final CI Result

97/97 tests executed, 97/97 passed. The torch smoke test ran on Linux (not skipped) and passed:

//bazel/tests:test_torch_smoke  PASSED in 12.2s
    PASSED  bazel.tests.test_torch_smoke.test_torch_importable (0.0s)
    PASSED  bazel.tests.test_torch_smoke.test_torch_tensor_basic (0.0s)

No libcudart.so.12 error. No libcublas.so error. Because the wheel is CPU-only — it literally doesn’t contain CUDA code. No lazy loading needed. No eager loading to crash. The CUDA question is simply not asked.

This is the actual final state of the PR.

The Journey, Compressed

Seven phases. The final approach takes the best idea from Phase 2 (CPU-only wheels) and the key constraint from Phase 5 (single lockfile entry), then solves both with post-processing instead of fighting the resolver. The difference between Phase 2 and Phase 7 isn’t ambition — it’s knowing exactly which part of the toolchain to work around and which to leave alone.

The Lessons

“Simple” Is a Hypothesis, Not a Property

Option A was supposed to be the simple one. “Just remove the gate.” Three words. One line change. Five minutes.

The final diff includes a lockfile post-processing script, a patch to aspect_rules_py, nvidia-* overrides, and a cache purge. It looks manageable in the PR. The commit history tells a different story.

The simplicity of the final state is not evidence that the problem was simple. It’s evidence that the problem was explored thoroughly enough to find the manageable solution hiding inside the complex one.

Know Your Blast Radius

The Phase 4 revert was the most expensive lesson. Changing pyproject.toml in a monorepo doesn’t just affect your feature. It affects uv sync for every developer and every CI job. The darwin gate wasn’t just “a constraint for Bazel” — it was load-bearing for the Linux CI pipeline.

Before changing a shared config file, ask: “who else reads this file, and what happens to them when it changes?” In a monorepo, the answer is “everyone” and “probably something bad.”

Dead Ends Are Data

The CPU index approach (Phases 2 and 5) produced four patches to aspect_rules_py, a deep understanding of PEP 440 local version suffixes, and a catalog of aspect_rules_py parser bugs. All of that was “wasted” — the patches were reverted, the approach abandoned.

Except it wasn’t wasted. The CPU index work proved that dual-version lockfiles are fundamentally fragile with current tooling. That’s not something you can know from a design doc. You have to try it, watch it break in four different ways, and internalize why it breaks. The dead end was the proof that the other path is correct. And the URL-decode patch from Phase 2? That one survived all the way to Phase 7. The work wasn’t wasted — it was early.

Test on the Platform That Matters

The Phase 6 “lazy CUDA” theory is the most instructive failure in this whole saga. Here’s the chain of reasoning that went wrong:

1. PyPI torch on macOS imports without CUDA errors ✓ (true)
2. Therefore, CUDA is lazy-loaded ✗ (wrong inference)
3. Therefore, PyPI torch on Linux will also import without CUDA errors ✗ (wrong conclusion)

The macOS wheel doesn’t lazy-load CUDA. It doesn’t load CUDA at all, because it doesn’t have CUDA. Apple dropped CUDA support. The macOS PyPI wheel is already CPU-only. Testing import torch on macOS tells you exactly nothing about what happens on Linux.

This is “works on my machine” with a neuroscience-flavored rationalization on top. I didn’t just fail to test on Linux — I constructed a plausible theory for why I didn’t need to. The theory was elegant. The theory was wrong. The CI machine didn’t care about my theory.

If you’re solving a Linux problem, test on Linux. If you can’t test on Linux, don’t claim you’ve solved the Linux problem. The most dangerous test result is the one that confirms your hypothesis on the wrong platform.

The Most Expensive Assumption Is the “Obviously True” One

I could have tested import torch on a Linux CPU runner in Phase 1 and known immediately that the CUDA-bundled wheel crashes. That would have sent me straight to “we need CPU-only wheels for Linux” without the detour through Phase 6’s false confidence.

But I assumed it would crash (correct), then later assumed it wouldn’t (incorrect), and both times the assumption felt obvious enough that I didn’t question it. The most expensive assumption in engineering is the one that’s “obviously true” — you skip the verification step precisely because it seems unnecessary.

Trade-offs Are Fractal

The success levels post laid out four options (A, B, C, D). Each option seemed like one decision. In practice, Option A alone contained seven sub-approaches, each with its own trade-off tree:

• Standard PyPI vs CPU index
• Single version vs dual version
• Patched tooling vs clean tooling
• nvidia-* blocked vs nvidia-* resolved
• override-dependencies vs tool.uv.sources
• Lockfile as source of truth vs lockfile as post-processed artifact

The design doc says “Option A: add Linux wheels to lockfile.” The implementation says “which wheels, from which index, with which version scheme, resolved by which mechanism, blocking which transitive deps, post-processed how, verified by which test, on which platform?” Every level of zoom reveals more decisions.

What Changed (Final)

Nine files. If you only read the diff, it looks like a focused afternoon’s work. If you read the commit history, it’s a week of wrong turns, a false victory, and the realization that the right answer was hiding between two approaches that each failed on their own.

Where This Sits

Compared to the other options:

Option A trades a maintenance script for a clean dependency graph. Option B trades manual dependency tracking for a clean lockfile. Option C trades Bazel isolation for zero lockfile drama. Pick your maintenance burden.

Previous in series: What Does Success Look Like? | Option B: The Five Obstacles | Option C: System Torch in Docker