Bazel Torch: Making Bazel See System Torch in Docker

TL;DR: Bazel’s hermetic Python can’t see system-installed packages. Torch is 700MB and system-installed inside Docker. We built a two-layer escape hatch — local_runtime_repo for the interpreter, py_venv_test with include_system_site_packages=True for runtime isolation — so Bazel tests can reach torch without downloading it. The implementation took five CI iterations, each failing for a completely different reason.

This is the implementation follow-up to the series. Part 1 was observational. Part 2 was the scoreboard. Part 3 explained the resolution pipeline. The success levels post defined what “done” looks like. This post is the one where we actually build something — and then watch it break five times.

The Problem, Restated

In the dependency resolution post, I laid out four options for getting torch into Bazel’s dependency graph on Linux:

Option C is a deliberate trade: we give up Bazel’s package-level hermeticity and replace it with Docker’s environment-level hermeticity. The container IS the hermetic boundary — it’s just drawn at a different level than Bazel expects.

The Two Layers You Need to Break Through

Here’s the thing nobody tells you: getting Bazel to use a system-installed Python package requires punching through two layers of isolation. Miss either one and it silently doesn’t work.

Layer 1: The Interpreter (rules_python)

Bazel, by default, downloads its own Python interpreter. A hermetic, standalone binary that has no idea what’s installed on the host system. Your Docker container could have torch, tensorflow, and the complete works of Shakespeare installed in site-packages — Bazel’s Python doesn’t care. It brought its own.

The fix is local_runtime_repo from rules_python. It tells Bazel: “don’t download a Python. Use this one.”

# MODULE.bazel
local_runtime_repo(
    name = "system_python3",
    interpreter_path = "/usr/local/bin/python3",
    on_failure = "skip",  # gracefully degrade on non-Docker machines
)
register_toolchains("@local_toolchains//:all")

That on_failure = "skip" is important. On a macOS dev laptop, /usr/local/bin/python3 might not be where you think it is, or it might not have torch. The skip means Bazel falls back to its hermetic interpreter. The Docker environment opts in; everything else opts out. No if/else in the Bazel config, no platform switches — just graceful degradation.

Layer 2: Runtime Isolation (aspect_rules_py)

Here’s where I wasted an afternoon.

You’d think registering the system Python is enough. It’s not. Even when Bazel uses the Docker container’s Python interpreter, it runs it with the -I flag. That’s Python’s “isolated mode” — it strips site-packages from the import path. Your system Python has torch? Great. The -I flag makes it invisible.

This is py_test’s default behavior and you can’t change it. py_test always runs isolated. It’s a philosophical commitment to hermeticity baked into the rule itself.

The escape hatch is py_venv_test from aspect_rules_py:

# BUILD.bazel
load("@aspect_rules_py//py/private/py_venv:defs.bzl", "py_venv_test")

py_venv_test(
    name = "test_torch_smoke_system",
    srcs = ["test_torch_smoke_system.py"],
    include_system_site_packages = True,
    tags = ["manual"],
    target_compatible_with = ["@platforms//os:linux"],
)

What does include_system_site_packages = True actually do? Two things:

1. Strips the -I flag. Deep in py_venv.bzl (line 27, if you’re curious): args = [it for it in args if it not in ["-I"]]. That’s it. One list comprehension standing between you and your system packages.

2. Writes pyvenv.cfg with include-system-site-packages = true. This tells the venv to inherit from its base interpreter’s site-packages. Since the base interpreter is the Docker container’s Python (thanks to Layer 1), the venv inherits torch.

Both layers needed. Layer 1 gives you the right interpreter. Layer 2 lets that interpreter see its own packages.

Without Layer 1:  Bazel's hermetic Python → no torch anywhere
With Layer 1:     Docker's Python → torch in site-packages → but -I flag hides it
With Both:        Docker's Python → torch in site-packages → -I removed → import torch works

The Implementation (30 Minutes)

The actual code change was small. A MODULE.bazel addition, a new BUILD.bazel target, and a smoke test:

# bazel/tests/test_torch_smoke_system.py
import torch

def test_torch_import():
    """Verify torch is importable from system site-packages in Bazel."""
    assert torch.__version__

def test_tensor_creation():
    """Verify basic torch operations work."""
    tensor = torch.zeros(2, 3)
    assert tensor.shape == (2, 3)

Nothing fancy. import torch, create a tensor, assert it works. The test isn’t testing torch — it’s testing the plumbing.

The CI Adventure (Five Hours)

The implementation took 30 minutes. Making CI pass took five iterations over five hours. Each failure was a completely different kind of wrong. This is the part of the blog post where you go “oh, that’s why this was hard.”

Round 1: The Formatting Police

What failed: Buildifier format check.

Why: MODULE.bazel needs attributes in alphabetical order. I had them in logical order, like a person who cares about readability. Buildifier does not care about readability. Buildifier cares about alphabetical order.

Also, importing from @aspect_rules_py//py/private/py_venv:defs.bzl triggers a bzl-visibility warning — you’re reaching into a private directory. The warning is treated as an error in CI. Fix:

# buildifier: disable=bzl-visibility
load("@aspect_rules_py//py/private/py_venv:defs.bzl", "py_venv_test")

One line of code telling the linter “yes, I know what I’m doing.” Classic.

Round 2: The Ghost Failure

What failed: Pre-commit checks showed “All pre-commit checks passed” but the job was marked failed.

Why: Tailscale cleanup in a post-job hook threw an error. Nothing to do with my code. Nothing to do with pre-commit. A VPN cleanup script in CI infrastructure failed, and GitHub Actions treats post-job failures as job failures.

I stared at this for 20 minutes before scrolling past the green “All checks passed” output to find the red Tailscale error at the very bottom. Infrastructure flake. Moving on.

Round 3: The “Easy Way Out” Problem

What failed: test_torch_smoke_system ran in bazel test //... on regular CI — a runner with no Docker, no torch, no system site-packages. It failed because… of course it failed. There’s no torch.

First fix: Added tags = ["manual"] to skip it in wildcard //... runs. This works — manual means “only run when explicitly targeted.”

The problem with the fix: Now both torch smoke tests were being skipped. The macOS one (constrained to target_compatible_with = macos, running on a Linux runner — skip). The system one (tagged manual — skip). My test matrix was testing exactly nothing.

I briefly tried to claim victory. The user — me, in the other window — was not having it. Direct quote: “You’re taking the easy way out.” Which… fair. Skipping the test you wrote to prove a thing works is not a proof of anything. Back to the drawing board.

Round 4: The .dockerignore Ambush

New plan: Add a CI job that builds a Docker image with torch and runs the Bazel test inside it. Write a Dockerfile.torch-test, add a bazel-torch-docker job to bazel_build_test.yml, done.

The Dockerfile:

FROM python:3.12-slim
RUN pip install torch --index-url https://download.pytorch.org/whl/cpu
# ... install bazelisk, uv, etc.
COPY . .
RUN bazel test //bazel/tests:test_torch_smoke_system

What failed: no such package 'bazel/tests'.

Why: COPY . . respects .dockerignore. The repo’s .dockerignore had **/tests/ in it — a glob that strips every tests/ directory from the Docker context. Including bazel/tests/BUILD.bazel. The test target doesn’t exist because the test file was never copied into the image.

This is the kind of bug that nobody googles until they hit it. You’re staring at a file that clearly exists on disk, the Dockerfile clearly copies COPY . ., and Bazel says the package doesn’t exist. The .dockerignore is 50 lines long and the offending pattern is buried in the middle.

Round 5: Volume Mount (The One That Worked)

The .dockerignore problem revealed a deeper design issue: COPY . . means the Docker image is rebuilt every commit. Even if only a Python file changed, Docker invalidates the COPY . . layer and everything after it. Not cacheable. Not smart.

Better idea: the Dockerfile only sets up the environment. Python, torch, bazelisk, uv. No repo code. CI volume-mounts the checkout into the container at runtime:

# In CI workflow
- run: |
    docker build -f bazel/tests/Dockerfile.torch-test -t torch-test .
    docker run -v "${{ github.workspace }}:/workspace" -w /workspace \
      torch-test bazel test //bazel/tests:test_torch_smoke_system

Benefits:

• Docker image is stable. Only changes when Python version or torch version changes. Cacheable across commits.
• No .dockerignore issues. The repo isn’t copied into the image — it’s mounted. The full checkout is visible.
• Matches the production pattern. Our real training containers mount code too. The test environment mirrors reality.

Green check. Finally.

What CPU-Only Torch Buys You

Small detail that matters: the Dockerfile installs torch from https://download.pytorch.org/whl/cpu. That’s the CPU-only variant. ~200MB instead of 2GB+ for the CUDA version.

This is fine because the smoke test doesn’t need a GPU. It tests import torch and torch.zeros(2, 3) — both CPU operations. The point isn’t to test CUDA. The point is to test the plumbing: can Bazel’s Python, inside Docker, actually see torch?

If someone later wants a GPU smoke test, that’s a different Docker image on a different runner. The architecture supports it — you’d just write another Dockerfile with the CUDA torch and a GPU-capable CI runner. But for proving the two-layer escape hatch works, CPU torch is cheaper and faster.

The Scoping Trick: Explicit Opt-In

The whole system is explicitly opt-in at every level:

Six layers of “you have to mean it.” A regular py_test target sees nothing. A regular py_venv_test sees nothing. You need all six knobs turned on for system torch to be visible. This is the right level of paranoia for a hermeticity escape hatch.

The Philosophy (Briefly)

We didn’t break Bazel’s hermeticity. We moved the hermetic boundary.

Traditional Bazel: the build system is the hermetic container. Every dependency is fetched, checksummed, sandboxed. Nothing from the environment leaks in.

Our approach: the Docker container is the hermetic container. The Dockerfile pins Python 3.12, pins torch to a specific version, pins everything. It’s reproducible. It’s isolated. It’s just that the isolation happens at the Docker layer, not the Bazel layer.

This is a scoped trade. The vast majority of targets still use Bazel’s hermetic Python with @pypi// dependencies. Only the targets that explicitly opt into py_venv_test + include_system_site_packages break through. And those targets only run inside Docker, where the environment IS controlled.

The heresy isn’t that we let the environment in. It’s that we admit torch doesn’t fit the Bazel model and build a clean, explicit escape hatch instead of pretending the problem doesn’t exist.

What’s Next

This proves the mechanism for exactly one smoke test. The real work:

1. Migrate actual torch-using packages. The ~9 packages that genuinely import torch for GPU work. Each one gets a py_venv_test target alongside its existing py_test targets.

2. Test classification. The test classification post laid out the problem — ~40% of tests import torch but only need CPU. Now that we have the Docker plumbing, we can route those tests to cheaper runners.

3. GPU CI. A Dockerfile.torch-test-gpu with CUDA torch on GPU-capable runners. Same architecture, different image.

4. flash-attn, triton, and friends. Same escape hatch pattern, different packages. Whatever we built for torch generalizes — that was the whole point.

The two-layer model (interpreter selection + runtime isolation) is the foundation. Everything else is wiring.

This is the implementation post in the Bazel + GPU series. The full series: Part 1: When GPU Showed Up | Part 2: The Scoreboard | Part 3: The Resolution Pipeline | Success Levels | Test Classification. All from a real 95-package production monorepo.