Bazel Torch: Two PRs, One Compiler, and a Package That Didn’t Need Either

There’s a story arc in our Bazel migration that I’ve been writing about in pieces but never connected. It spans three months, two competing PRs, five failed attempts at hermetic compilation, a CUDA package with 80+ C++ source files that needed exactly zero lines of CC toolchain config, and now a vendor fork of NCCL that compiles in a completely different CI pipeline and gets installed via apt.

The punchline is that we spent weeks debating how to give Bazel a C++ compiler, and the biggest native-code packages in our monorepo don’t use a C++ compiler through Bazel at all. The problems looked identical from the outside. They’re completely orthogonal.

If you’ve been following the series, this is the connective tissue. If you haven’t – the previous posts cover the technical depth. This one is the map.

The timeline

Four different problems wearing the same hat: “Bazel and C++ code.” Four completely different solutions.

Problem 1: fasttext needs a compiler at pip-install time

fasttext ships as an sdist – source code that needs gcc or clang to compile. When Bazel’s sdist_build rule tries to compile it in a hermetic sandbox, the compiler isn’t there. Build fails. This is the most straightforward version of “Bazel meets C++.”

Two people solved this at the same time, in different ways.

Ryan’s solution: the one that shipped

PR #21967. Merged March 3.

Three changes:

1. Patch aspect_rules_py’s sdist_build rule to add use_default_shell_env=True
2. Add --action_env=PATH, --action_env=CC, --action_env=CXX to .bazelrc
3. Add pip_annotations.toml declaring fasttext’s build-time deps
4. Remove fasttext from the Bazel exclusion list

The host’s compiler leaks into the sandbox through environment variables. setup.py finds gcc. fasttext compiles. Done.

It works. It also has a known limitation – Ryan put a TODO on it himself:

TODO(DEVEX-418): replace with a Bazel-managed cc_toolchain for hermeticity.

The --action_env=PATH in .bazelrc is the part that stings. PATH values differ between machines. Different PATH means different cache keys, which means cache misses. On a monorepo where remote caching saves us hours per day, that’s not nothing.

Although – and this is the part I got wrong initially – --action_env=PATH doesn’t pollute cache keys for all actions. Only for actions that use use_default_shell_env=True. Which is just sdist_build. The blast radius is smaller than I claimed. More on that in a sec.

My solution: the one that didn’t ship

PR #22578. Still open.

The plan was simple: register a hermetic LLVM/Clang compiler via toolchains_llvm, wire it into sdist_build, remove the --action_env hacks entirely. No host compiler leak. Same compiler on every machine. The Bazel Way.

Five attempts. Five failures. Each one peeled back a layer of something I didn’t understand.

I wrote the full technical walkthrough already, but the short version:

1. Hermetic toolchain – find_cc_toolchain() returns a truthy object even when all_files is None. Null check passes, .to_list() blows up.
2. Add sysroot – sysroot flag only gets injected into CppCompileAction. sdist_build runs python -m build as a subprocess. Flag never reaches the compiler.
3. Extract flags via cc_common API – Got the right flags, but they reference paths relative to Bazel’s execroot. And then…
4. Simplified flags – Docker CI passed. Linux CI didn’t. Same root cause as #3.
5. Gave up on full hermetic – fell back to use_default_shell_env=True, same as Ryan’s core fix.

The root cause is architectural. build_helper.py in aspect_rules_py changes CWD to a temp directory before running python -m build. Everything in toolchains_llvm – the compiler wrapper, the include paths, the linker scripts – uses paths relative to Bazel’s execroot. Change the CWD and every relative path breaks.

There’s an 8-line fix that would bridge them:

EXECROOT = os.getcwd()  # capture before CWD changes
for var in ("CC", "CXX"):
    val = env.get(var, "")
    if val and not os.path.isabs(val):
        env[var] = os.path.join(EXECROOT, val)

But that means patching a second file in aspect_rules_py. I stopped at one patch.

Why is the CC toolchain hard?

I need to correct something I’ve been implying. The CC toolchain is not hard in general. Registering toolchains_llvm or rules_cc in a Bazel project that builds cc_library or cc_binary targets is… fine. Normal. Well-documented. You register the toolchain, you point it at a compiler, Bazel drives everything. Thousands of projects do this.

The CC toolchain is hard for our specific problem because of an impedance mismatch between four layers that each have incompatible assumptions about how compilation works.

Here’s the compilation flow for an sdist build:

Bazel's sdist_build rule
  -> runs: python -m build (subprocess)
    -> which runs: setup.py
      -> which calls: gcc/clang (to compile C extensions)

Notice what’s happening. Bazel isn’t calling the compiler. Python is. Bazel starts a subprocess that starts a build tool that starts a compiler. Three levels of indirection. And each layer has its own ideas about where the compiler lives, how paths work, and what the current directory means.

Here’s the impedance mismatch table – the reason my PR has five attempts in it:

Every attempt broke at the boundary between two of these layers:

1. Just register the toolchain – setup.py can’t see it. Hermetic sandbox, no PATH, no CC env var. The toolchain exists in Bazel’s world but Python doesn’t know about Bazel’s world.
2. Pass the compiler path as CC env var – the path is relative to execroot. Works as long as you’re standing in the execroot.
3. Make it absolute – build_helper.py changes CWD to a temp directory before running the subprocess. The compiler wrapper (cc_wrapper.sh) still uses relative paths internally for sysroot and includes. Breaks.
4. Pass only the sysroot flag – works in Docker (where the sysroot layout matches). Fails on regular Linux (different sysroot layout, different include paths).
5. Give up on hermetic, use use_default_shell_env=True – same as Ryan’s fix. The host compiler leaks in. It works.

The fix is those 8 lines: capture the execroot path before build_helper.py changes CWD, absolutize the compiler paths. But that’s a patch to a second file in aspect_rules_py, on top of the patch we already carry for use_default_shell_env. I stopped at one patch. There’s a limit to how many upstream files you want to fork-maintain.

For normal cc_library targets – where Bazel directly invokes the compiler – none of this matters. The impedance mismatch only exists because sdist_build runs Python-as-a-subprocess-that-runs-a-compiler. It’s a Rube Goldberg chain of build tools, and the CC toolchain breaks at every joint.

The honest comparison

Here’s the table I should have written from the start, before I made claims about Ryan’s approach being worse than mine:

The core fix is the same. One line: use_default_shell_env = True on the sdist_build action. Everything else is scaffolding.

I wrote a PR description arguing that Ryan’s --action_env=PATH “pollutes all cache keys.” Then I read the Bazel source code and discovered that --action_env only manifests in actions with use_default_shell_env=True. On Linux, PATH is already in SHELL_ACTION_ENV by default. Ryan’s --action_env=PATH is a no-op on Linux.

My PR adds a hermetic LLVM toolchain that sdist_build doesn’t even use (because of the CWD problem). Ryan’s PR adds --action_env flags that are mostly redundant on Linux. We were both carrying dead weight. His was smaller.

The only concrete difference: my PR adds a test_fasttext_smoke test that validates the build end-to-end. That’s genuinely useful. Everything else was me overengineering a solution to a problem Ryan had already pragmatically solved.

What I actually learned about being wrong

The correction post was uncomfortable to write. I made technical claims in a PR description – about Bazel internals, about cache key semantics, about the scoping of --action_env – that didn’t hold up when I read the source code. Not “slightly off.” Wrong.

Three things were wrong:

1. "–action_env=PATH pollutes all actions" – only affects actions with use_default_shell_env=True
2. “use_default_shell_env=True only leaks PATH” – also leaks LD_LIBRARY_PATH, LC_CTYPE, and on macOS, SDKROOT
3. “Our approach scopes the leak, Ryan’s is global” – both PRs set use_default_shell_env=True on the same action. The “leak” is identical.

The meta-lesson: don’t reason about implementation behavior from documentation. Read the implementation. Specifically, SpawnAction.java, ActionEnvironment.java, and BazelRuleClassProvider.java told me everything. The docs told me a story that was close enough to be dangerous.

Problem 2: mai_kernels has 80+ C++ files and needs zero CC toolchain

This is where the story gets interesting.

While the CC toolchain debate was happening, I looked at mai_kernels – the package everyone assumed was permanently stuck in the “Blocked by Non-Python Code” column. Nine pybind11 native modules. CMake/nvcc build pipeline. CUDA, CUTLASS, MathDx, MPI. The works.

The assumption was: “this package has C++ code, so we need a CC toolchain to migrate it to Bazel.”

The assumption was wrong.

mai_kernels doesn’t need a CC toolchain because Bazel doesn’t compile its C++ code. Nobody asked Bazel to. The pretrain_base Docker image already compiles all nine .so modules during uv sync via scikit-build-core -> CMake -> nvcc. Those binaries exist at runtime inside the container.

Bazel’s job is just managing the Python source layer. The BUILD file is surprisingly small:

py_library(
    name = "mai_kernels",
    srcs = glob(["src/mai_kernels/**/*.py"], exclude = ["src/mai_kernels/**/tests/**"]),
    imports = ["src"],
    deps = ["//lib/mai_nano/src/mai_nano/mocks", "@pypi//numpy"],
)

Two deps. numpy and the mock utilities. The nine native .so files, torch, triton, CUDA libraries – all come from system site-packages via include_system_site_packages=True. Same pattern as system torch.

And here’s the part that makes it clean: mai_kernels/__init__.py already uses magic_mock_packages(). When the native .so files aren’t available – like on CPU – the modules get mocked. The py_library builds and type-checks fine on CPU. Only the tests need GPU. The CPU/GPU boundary was already designed into the package.

I wrote the full migration story separately. The point for this post is that mai_kernels and fasttext look like the same problem from the outside – “Bazel and C++ code” – but they’re completely different problems with completely different solutions.

Problem 3: NCCL gets a vendor fork and Bazel still doesn’t care

Just as I was wrapping up the mai_kernels migration, Ryan opened DEVEX-510 – “setup forked build of nccl.” And it’s yet another native-code story where the answer to “do we need a CC toolchain?” is “no.”

NCCL – NVIDIA’s Collective Communications Library – is the glue that coordinates GPU-to-GPU communication in distributed training. It’s C and CUDA under the hood. The upstream team wants a fork so they can carry internal bug fixes and patches without waiting for NVIDIA releases.

Here’s the plan:

1. Fork NCCL to infinity-microsoft/mai-nccl (DEVEX-511)
2. Build it using NCCL’s own Makefiles (make + nvcc) in a GitHub Actions workflow
3. Package the output as a .deb
4. Upload via gh release create + gh release upload
5. In the Docker image build: gh release download + apt install

That’s it. The entire compilation lifecycle lives inside a GHA workflow that has nothing to do with Bazel. The resulting .deb gets installed into the Docker image like any other system package. By the time Bazel runs, NCCL is just another shared library sitting in /usr/lib. Invisible.

This is the third variation of the same pattern. mai_kernels compiles in the Docker image build itself (CMake/nvcc during uv sync). torch is pre-compiled upstream. NCCL will be compiled in a dedicated GHA workflow. Three different “when” and “where” answers – same Bazel outcome: don’t touch it, it’s already there.

What makes NCCL interesting is how far removed the compilation is from anything Bazel-adjacent. It’s not even in the same repository. A separate repo, a separate CI pipeline, a separate artifact format (.deb), installed by the system package manager. The distance between “this code gets compiled” and “Bazel sees the result” is about as wide as it can get.

The matrix

Here’s the distinction that took me three months to internalize. It’s not a two-by-two – it’s a “where does compilation happen?” question with three answers:

One of these needs a CC toolchain. The other three are already compiled before Bazel even starts. Different compilation pipelines, different lifecycles, same “but it has C++ code” surface appearance.

fasttext = sdist that needs gcc at pip install time -> CC toolchain or use_default_shell_env -> Ryan’s PR / my PR

mai_kernels = CUDA code compiled in Docker via CMake/nvcc -> already done by image build -> py_library + system site-packages -> PR #23478

torch = pre-compiled in Docker image -> not even compiled by us -> system site-packages -> PR #21169

NCCL fork = compiled in a separate GHA workflow, packaged as .deb, installed into Docker via apt -> Bazel never sees it

If you try to solve a Docker-compiled problem with a CC toolchain, you’ll spend weeks rewriting 418 lines of CMakeLists.txt into rules_cuda for zero practical gain. If you try to solve an sdist problem with system site-packages, it won’t work because the compilation hasn’t happened yet. And if your native code is compiled in an entirely separate CI pipeline and installed as a system package? You don’t even need to think about it.

The solution depends on when and where the compilation happens. Not on whether C++ is involved.

Where we are now

The key lesson: we made the assumption mistake with mai_kernels. Looked at “CUDA/C++ code” and assumed it needed a CC toolchain before actually looking at what the code does. It didn’t. Don’t repeat that with lib/bus – investigate first, read the build files, then pick an approach. Could be a cc_library rewrite, a rules_foreign_cc wrap, or the Docker route. Depends entirely on what the CMakeLists.txt looks like and what it depends on.

Ryan’s TODO still stands: DEVEX-418, replace --action_env with a managed CC toolchain. My PR (#22578) is the half-built attempt. The CWD problem in build_helper.py is the real blocker – and it might be worth an upstream PR to aspect_rules_py rather than carrying a second local patch.

When do you actually need a CC toolchain?

After four packages and three months, here’s the decision tree I wish I’d had on day one:

Step 1: Does the package have native code (C, C++, CUDA, Fortran, Rust)?

No -> Normal py_library. You’re done.

Yes -> Continue.

Step 2: Is that native code already compiled before Bazel runs?

This is the only question that matters. Check these, in order:

• Pre-compiled upstream? (torch, numpy wheels, etc.) -> No CC toolchain. Use pre-built wheels or system site-packages.
• Compiled in the Docker image build? (mai_kernels via CMake/nvcc during uv sync) -> No CC toolchain. py_library + include_system_site_packages=True.
• Compiled in a separate CI pipeline? (NCCL fork via GHA workflow -> .deb -> apt install) -> No CC toolchain. It’s a system package by the time your container starts.
• Compiled at pip install time from an sdist? (fasttext, pyyaml) -> Yes, you need a CC toolchain (or use_default_shell_env to leak the host compiler).

That last case – sdist packages compiled inside Bazel’s sandbox – is the only case where a CC toolchain matters. It’s also the narrowest case. Most native code in a Python monorepo falls into one of the first three buckets.

The instinct is to see “C++ code” and reach for a CC toolchain. Resist it. Ask “where does this get compiled?” first. Four out of four times so far, the answer has been “somewhere else” for everything except sdists.

The CC toolchain beyond sdist

The CC toolchain currently serves exactly one consumer: fasttext’s sdist build. Zero cc_library targets in the monorepo. All C++ code is compiled somewhere else – Docker, upstream, GHA. Bazel has never called a C++ compiler for our code.

But that won’t last:

The big one is lib/bus. But the approach is: investigate first. Read the build files. Check the complexity spectrum. If it’s a grocery-list CMakeLists.txt, cc_library rewrite or rules_foreign_cc wrap. If it’s got exotic deps, Docker route. We don’t know yet, and that’s fine – the mistake would be assuming before looking.

The honest framing: the CC toolchain is either infrastructure-for-the-future or over-engineering-for-the-present. Which one depends on what we find when we actually open lib/bus’s build files.

Why C++ is uniquely hard in Bazel

Here’s something I didn’t fully appreciate until deep into this migration: C++ is categorically different from every other language Bazel supports. Not harder. Different.

Go has go build. Python has pip. Rust has cargo. Java has javac. Each language has ONE canonical build tool. When Bazel adds support, there’s a standard to translate from. rules_go takes Go’s one build model and maps it to Bazel’s world. The translation is mechanical. Boring, even.

C++ was created in 1979. C in 1972. There has never been a standard build tool. Instead, the ecosystem produced dozens of them over fifty years: Make (1976), Autoconf (1991), CMake (2000), Ninja (2012), Meson (2013), Visual Studio, and… Bazel itself.

There is no cpp build command. Never was.

This means:

• rules_go TRANSLATES Go’s one standard build model into Bazel
• rules_cc IS a build model for C++. One of many. It doesn’t translate anything – you write the build instructions from scratch in Bazel’s language.

rules_cc tells Bazel “compile these files, link these deps.” That’s it. The CC toolchain tells it WHICH compiler to use. Two separate concerns.

This distinction explains why wrapping C++ in Bazel is so much harder than wrapping Go or Rust. For those languages, there’s a canonical source of truth to import from. For C++, every project invented its own truth.

rules_cc vs rules_foreign_cc

“Doesn’t rules_cc teach Bazel how to build C++?” Yes. But only for code described in Bazel’s own language. If the project uses CMake, Make, Autoconf, or anything else – rules_cc doesn’t help. That’s what rules_foreign_cc is for.

rules_cc = Bazel is the C++ build system. You write cc_library, cc_binary, cc_test in BUILD.bazel. Bazel calls the compiler, manages paths, tracks deps, caches objects. Full control, full commitment. You’re rewriting the project’s build in Bazel’s language.

rules_foreign_cc = Bazel runs another C++ build system. You write cmake_external in BUILD.bazel. Bazel says “run CMake inside my sandbox and give me the output.” The CMakeLists.txt stays untouched. Bazel orchestrates the subprocess and captures artifacts.

rules_cc teaches Bazel to be a C++ build system. rules_foreign_cc teaches Bazel to run one.

The CMakeLists.txt complexity spectrum

Whether you can rewrite, wrap, or have to bail depends entirely on what the CMakeLists.txt actually does. It’s not one thing on the complexity spectrum. It IS the spectrum:

rules_foreign_cc shines when the CMake project is self-contained – “compile these C files with a C compiler.” It breaks down when:

• The project needs exotic deps in the sandbox (CUDA toolkit, torch headers, MPI)
• It does FetchContent (network access inside Bazel’s sandbox = no)
• It does hardware detection (execute_process(COMMAND nvcc --version))

For mai_kernels, even rules_foreign_cc doesn’t save you – the dependency chain is too exotic for any sandbox. Docker is genuinely the only option.

The full spectrum

Simple C files ──→ Simple CMake ──→ Complex CMake ──→ CUDA/exotic CMake
     |                  |                 |                    |
  cc_library        cc_library       rules_foreign_cc      Docker builds it
  (rewrite)         (rewrite)        (IF deps are simple)  (only real option)
                         or                  or
                    rules_foreign_cc   Docker builds it

Where you land on this spectrum determines your approach. And you figure that out by reading the CMakeLists.txt, not by guessing.

The pattern

Every native-code package in a Python monorepo falls into one of four buckets:

1. Sdist with C extensions (fasttext, pyyaml) – needs a compiler at pip install time. Fix: CC toolchain or use_default_shell_env. Small blast radius, tractable problem. This is the only bucket that needs a CC toolchain.

2. CUDA/GPU code compiled in Docker (mai_kernels) – compiled during image build via CMake/nvcc/scikit-build-core. Fix: py_library + include_system_site_packages. Bazel manages Python, Docker manages native code.

3. Pre-compiled upstream (torch, triton, numpy) – someone else compiled it. Pre-built wheel or system package in Docker. Fix: system site-packages. Bazel doesn’t need to know.

4. Compiled in a separate CI pipeline (NCCL fork) – your org compiles it, but not in Bazel and not in the Docker image build. GHA workflow -> artifact -> system package install. Fix: nothing. It’s already there by the time anything runs.

Buckets 2, 3, and 4 all converge on the same answer: “Docker handles it.” The compilation happens somewhere upstream of Bazel – whether that’s in the Dockerfile, in PyTorch’s release pipeline, or in a dedicated GHA workflow. By the time Bazel starts, the native code is just shared libraries on disk. Invisible.

The first question isn’t “do we need a CC toolchain?” The first question is “when does this code get compiled, and where?” If the answer is anything other than “inside Bazel’s sandbox at pip-install time,” you don’t need Bazel to know about C++ at all.

Two teammates. Two PRs. One compiler debate. And three native-code packages that watched from the sidelines, already knowing the answer was “none of the above.”

Previous posts

The full arc, in reading order:

• Three Ways to Smuggle torch into Bazel – the original torch experiments
• Making Bazel See System Torch Inside Docker – Option C implementation
• Why Bazel Can’t Compile Your Python C Extensions – sdist + CC toolchain explainer
• Stocking the Kitchen – the hermetic toolchain attempt (before I discovered it was wrong)
• I Was Wrong About use_default_shell_env – the correction post
• Migrating a CUDA Package Without Bazel Knowing About CUDA – mai_kernels
• Everything Below import torch – reference guide for the whole native build stack