Every other Python package in our monorepo downloads a wheel from PyPI and runs in Bazel’s hermetic sandbox. Numpy, pydantic, flask — they all go through the same pipeline. uv.lock resolves them, @pypi// makes them Bazel targets, tests run in a clean sandbox. Beautiful.
So why does torch need its own Python interpreter?
This is the question I kept getting when I presented the system torch PR. “Can’t we just… add torch to the lockfile like everything else?” The intuition is strong. The answer involves shared libraries, CUDA driver coupling, and the fact that torch is not really a Python package — it’s a C++/CUDA project that happens to have a Python frontend.
Start with what “hermetic” actually means
Bazel’s hermetic Python is a standalone interpreter binary, downloaded at build time, pinned to an exact version. It knows about exactly one thing: wheels that Bazel installed for it from uv.lock. It doesn’t see your system Python. It doesn’t see /usr/lib. It doesn’t see CUDA. It’s in a sandbox, and the sandbox is the whole point.
For pure Python packages — or even packages with pre-compiled wheels like numpy — this works perfectly. The wheel contains everything. Download it, hash-verify it, unpack it into the sandbox, done. The hermetic Python can import it, and nothing outside the sandbox matters.
Torch is different.
The lockfile problem (and why it’s a choice)
First issue: torch isn’t even in uv.lock on Linux. Our pyproject.toml has:
"torch >= 2.7.0; sys_platform == 'darwin'"
Torch is macOS-only in the lockfile. On Linux, uv.lock has no torch wheel to fetch. Bazel can’t install what doesn’t exist in the resolution.
But this is a choice, not a law of physics. We could remove the platform marker and let uv resolve a Linux torch wheel. I tried that — it’s Option A. macOS works. Linux breaks because of marker-combining issues between uv’s lockfile format and rules_py’s interpretation. Fixable eventually, but not today.
Okay, so suppose that gets fixed. Suppose we have a Linux torch wheel in the lockfile. Are we done?
No. And this is where it gets interesting.
The CUDA coupling problem
Our Docker CI images ship torch pre-compiled against specific CUDA drivers (12.x), cuDNN (9.x), and NCCL versions. These aren’t suggestions — they’re binary-level contracts. The torch .so files contain compiled GPU kernels targeting specific CUDA compute capabilities, linked against specific versions of libcudart.so and libcudnn.so.
The PyPI torch wheel is either CPU-only or compiled against a generic CUDA version. Use the wrong torch against the wrong CUDA stack and you get:
CUDA error: no kernel image is available for execution on the device
Or worse — it imports fine, passes basic tensor operations, and then fails deep inside a training loop when it hits a kernel that needs compute capability 9.0 but only has 8.0 compiled in. That kind of bug costs you a GPU cluster booking and half a day of debugging.
In production ML training, you need the exact torch that was built for your GPU stack. Not close. Not compatible. Exact. The Docker images are built that way on purpose — specific CUDA, specific cuDNN, specific NCCL, specific torch, all tested together.
A PyPI wheel doesn’t give you that.
The ABI problem (the one that actually kills you)
Here’s the part most people don’t think about. Torch is not a .py file collection. It’s a set of compiled C++ and CUDA shared libraries — .so files — with a Python wrapper on top. When you import torch, Python loads libtorch_python.so, libtorch_cuda.so, and a dozen other shared objects.
These .so files were compiled against a specific Python binary. Not “Python 3.12” in the abstract — a specific build of Python 3.12, with specific struct layouts, specific memory alignment, specific ABI flags. The system torch in our Docker image was compiled against the Docker image’s system Python.
Bazel’s hermetic Python is a different binary. Same version number, possibly. Same ABI? Not necessarily. And even “not necessarily” is enough to make ctypes segfault, make extension loading produce garbage, or make things work 99% of the time and then corrupt memory at the worst possible moment.
The key insight: a .so file is married to the Python binary it was compiled against. You can’t take a shared library built for one Python and load it into a different Python just because the version numbers match. It’s like trying to use a car key from the same model year — same shape, different notches.
“But what if we just…”
This is usually where someone proposes a clever workaround. Let me save you the debugging time.
“Add system site-packages to hermetic Python’s sys.path.” You could add /usr/local/lib/python3.12/site-packages to the hermetic Python’s import path. Python would find the torch package. Then it would try to load libtorch_python.so, which was compiled against the system Python’s ABI, not the hermetic Python’s ABI. You now have a Frankenstein interpreter — one Python’s brain running another Python’s compiled organs. This sometimes works by accident. When it doesn’t, the failure mode is a segfault, not an ImportError. Good luck debugging that in CI.
“Download the same torch version from PyPI.” Same version number doesn’t mean same binary. The PyPI wheel was compiled in a different build environment, against different CUDA libraries (or no CUDA at all), with different compiler flags. You’ve matched the version string but not the binary-level reality.
“Pass through CUDA library paths into the sandbox.” Even if you punched LD_LIBRARY_PATH holes in the sandbox so hermetic Python can find libcudart.so and libcudnn.so, you still have the ABI mismatch between the hermetic Python and the system torch’s .so files. The CUDA libraries are only half the problem. The Python extension ABI is the other half.
Every workaround crashes into the same wall: you cannot mix two different Python builds at the shared library level. You either use the system Python with the system torch, or you use a fully hermetic torch that was compiled specifically for the hermetic Python. There’s no middle ground.
The clean answer: local_runtime_repo
local_runtime_repo from rules_python does something simple: it tells Bazel “this system Python at this path IS a valid Bazel toolchain.”
local_runtime_repo(
name = "system_python3",
interpreter_path = "/usr/local/bin/python3",
on_failure = "skip",
)
No ABI mismatch — system torch was built for system Python, and Bazel is now using system Python. No path hacking — system Python already knows where CUDA lives, where cuDNN lives, where torch lives. No sandbox holes — py_venv_test creates a venv from system Python, and that venv naturally inherits system packages when you set include_system_site_packages = True.
The interpreter, the shared libraries, and the CUDA stack all came from the same Docker image, built together, tested together. You’re not smuggling anything across a boundary. You’re removing the boundary.
The mental model
This is how I think about it:
Hermetic Python (downloaded by Bazel):
→ Sees: PyPI wheels installed by uv from the lockfile
→ Can't see: system packages, CUDA, GPU drivers
→ Good for: everything except torch on Linux
System Python (pre-installed in Docker):
→ Sees: system torch, CUDA 12.x, cuDNN 9.x, NCCL
→ Registered via: local_runtime_repo
→ Good for: torch tests on Linux CI
Two interpreters, two roles, no overlap. The custom py_test macro routes tests to the right one:
- Linux torch tests → system Python + system torch (CUDA-enabled)
- macOS torch tests → hermetic Python + PyPI torch (CPU-only, which is fine)
- Everything else → hermetic Python, standard pipeline
On macOS, the PyPI torch wheel works great — it’s CPU-only, no CUDA needed, no ABI drama. Developers can run torch tests locally. It’s only on Linux CI, where we need the exact CUDA-compiled torch that ships in Docker, that we need the second interpreter.
“But you’re breaking hermeticity!”
Yes. For torch tests, we’re trading Bazel’s package-level hermeticity for Docker’s environment-level hermeticity. The container IS the hermetic boundary — it’s just drawn at a different level than Bazel expects.
The Docker image is pinned, versioned, and reproducible. Every CI run uses the same image with the same torch, same CUDA, same Python, same everything. It’s hermetic. It’s just hermetic at the container layer instead of the Bazel sandbox layer.
For a GPU ML monorepo, that’s the right tradeoff. The alternative — rebuilding torch from source inside Bazel with the exact CUDA SDK, matching the hermetic Python’s ABI, and teaching Bazel about GPU drivers — is technically possible in the same way that building a ship in a bottle is technically woodworking.
Why this matters beyond torch
Any Python package with compiled C extensions that link against system libraries has this same problem in Bazel. Torch is the biggest example, but it applies to anything that ships .so files compiled against a specific Python build and specific system libraries.
The pattern — register the system interpreter via local_runtime_repo, use py_venv_test with system site-packages for tests that need system dependencies — is general. Torch just happens to be the one where the consequence of getting it wrong is a GPU cluster producing garbage instead of gradients.
Two Pythons isn’t a hack. It’s the acknowledgment that some packages are inseparable from the system they were built on.