Bazel CC Toolchain: I Was Wrong About use_default_shell_env

2026/03/03

BUILDbazeldependenciespython

I spent a week trying to give Bazel a hermetic C++ compiler for building Python sdists. Made five attempts. Wrote a PR. Wrote claims about how Bazel’s environment model works. Then actually read the Bazel source code and discovered several of those claims were wrong.

This is the corrected version. I’m going to walk through what I tried, why it failed, and what I got wrong about the alternative approach. Every claim about Bazel internals cites source code — because apparently I can’t trust myself to reason about it from documentation alone.

The problem

We have a Python monorepo on Bazel 8.5.1 with aspect_rules_py v1.8.4 for dependency management. Most of our ~300 Python packages are wheels — pre-compiled, no drama. Then there’s fasttext.

fasttext wraps Facebook’s C++ text classification library. It ships as an sdist — source code that needs a C compiler to build. When Bazel’s sdist_build rule tries to compile it:

error: command 'gcc' failed: No such file or directory

Bazel’s sandbox doesn’t expose the host’s compilers. That’s the whole point of Bazel. So we need to either bring a compiler into the sandbox or let the sandbox peek outside.

Quick background

Skip to the attempts if you know these concepts.

Sdists vs wheels. A wheel is pre-compiled — unzip and done. An sdist is source code that needs a compiler, linker, headers, the works. Most popular packages ship wheels. Some — like fasttext — only ship sdists.

Bazel’s sandbox. Build actions run in isolation with only declared inputs. No /usr/bin, no $PATH, no system headers. This gives you reproducibility and caching. It also means setup.py can’t find gcc.

toolchains_llvm. A Bazel module that downloads a pre-built LLVM/Clang distribution and registers it as Bazel’s CC toolchain. Hermetic compiler — same version everywhere.

use_default_shell_env. A flag on ctx.actions.run() that passes a small set of host environment variables into the action. More on what “small set” actually means later — I got this wrong initially.

cc_wrapper.sh. When toolchains_llvm sets up the compiler, it generates a wrapper script with a hardcoded relative path:

toolchain_path_prefix="external/toolchains_llvm++llvm+llvm_toolchain_llvm/"

This path is relative to Bazel’s execroot — the staging directory for action inputs. Works great from the execroot. Breaks from anywhere else. Remember this.

Attempt 1: full hermetic toolchain

The plan: register toolchains_llvm, patch sdist_build to discover the CC toolchain via find_cc_toolchain(), pass compiler paths as CC/CXX environment variables.

cc_toolchain_info = find_cc_toolchain(ctx, mandatory=False)
cc_path = cc_common.get_tool_for_action(
    feature_configuration=feature_configuration,
    action_name="c-compile",
)
env["CC"] = cc_path

Hit two bugs.

Bug 1: find_cc_toolchain(ctx, mandatory=False) returns an object, not None. The truthiness check passes, but all_files is None. So .to_list() blows up:

'NoneType' value has no field or method 'to_list'

Bug 2: Even after guarding all_files, clang couldn’t find C++ standard library headers. fasttext’s setup.py runs a C++17 compile test and fails:

RuntimeError: Unsupported compiler -- at least C++17 support is needed!

The LLVM distribution has the compiler but not system headers. It expects them from a sysroot.

Result: Dead.

Attempt 2: add sysroot

Tell clang where system headers live:

llvm.sysroot(name = "llvm_toolchain", path = "/", targets = ["linux-x86_64"])

The sysroot flag gets injected by Bazel’s feature system when creating a CppCompileAction. But sdist_build doesn’t create a CppCompileAction — it runs python -m build as a subprocess. The sysroot is never passed to the compiler.

Result: Same C++17 error. Right idea, wrong layer.

Attempt 3: extract flags via cc_common API

If Bazel knows the right flags, extract them myself:

flags = cc_common.get_memory_inefficient_command_line(
    feature_configuration=feature_configuration,
    action_name="c++-compile",
    variables=compile_variables,
)

This is the pattern from rules_foreign_cc — extract Bazel’s CC flags and pass them to a non-Bazel build system. Got --sysroot=/, -stdlib=libc++, etc. Concatenated them into CFLAGS/CXXFLAGS.

Progress — but -stdlib=libc++ makes clang look for libc++ headers at a path relative to the LLVM distribution, which is relative to Bazel’s execroot.

And here’s the root cause.

aspect_rules_py’s build_helper.py creates a temp directory, copies the sdist source there, and runs python -m build with cwd set to that temp directory:

# build_helper.py (simplified)
with tempfile.TemporaryDirectory() as tmpdir:
    shutil.copytree(sdist_path, tmpdir)
    subprocess.run(["python", "-m", "build"], cwd=tmpdir)

When CWD changes to /tmp/bazel-sdist-build-xyz/, every relative path from the execroot breaks. The libc++ headers at external/toolchains_llvm++.../include/c++/v1/? Doesn’t exist relative to /tmp/.

Result: Headers not found. CWD change killed it.

Attempt 4: simplified flags

Skip the fancy flags. Just pass CC/CXX paths and --sysroot=/:

env["CC"] = cc_path
env["CXX"] = cxx_path
env["CFLAGS"] = "--sysroot=/"
env["CXXFLAGS"] = "--sysroot=/"

Docker CI: passed. Docker images have system Python and gcc, and the sysroot approach finds headers at /usr/include/.

Linux CI: failed. cc_path points at cc_wrapper.sh, which contains:

toolchain_path_prefix="external/toolchains_llvm++llvm+llvm_toolchain_llvm/"

When build_helper.py changes CWD, the wrapper tries to find clang at a relative path from /tmp/. Same root cause as attempt 3.

The realization: This is architectural. build_helper.py changes CWD. Everything in toolchains_llvm uses execroot-relative paths. These two designs are structurally incompatible.

There’s an 8-line fix that would bridge them:

# In build_helper.py — capture execroot before CWD changes
EXECROOT = os.getcwd()

# Before subprocess call — resolve relative compiler paths
env = os.environ.copy()
for var in ("CC", "CXX"):
    val = env.get(var, "")
    if val and not os.path.isabs(val):
        env[var] = os.path.join(EXECROOT, val)

This would let hermetic CC paths survive the CWD change. But it means patching build_helper.py — a second file in aspect_rules_py. I didn’t go this route for the initial PR.

Result: Docker green, Linux red. Root cause confirmed.

Attempt 5: use_default_shell_env = True

After four attempts, the minimal concession:

ctx.actions.run(
    # ... existing arguments ...
    use_default_shell_env = True,
)

Let the build action see the host’s environment variables. setup.py finds gcc via PATH.

The full patch also adds use_cc_toolchain(mandatory=False) and fragments=["cpp"] — this registers the rule as a CC toolchain consumer. It doesn’t do anything today since the actual compilation goes through the host compiler. It’s scaffolding for when the CWD issue gets fixed upstream.

Result: CI green. PR #22578 ships.

Where I was wrong

My teammate Ryan had a parallel PR (#21967) that also used use_default_shell_env=True in the sdist_build patch, plus --action_env=PATH, --action_env=CC, and --action_env=CXX in .bazelrc. In my PR description, I argued that my approach was better because his “--action_env=PATH breaks hermeticity for all actions.”

I was wrong about how this works. I went back and read the Bazel source code. Here’s what I found.

Claim: “–action_env=PATH pollutes all actions”

Wrong.

--action_env=PATH (the “inherit” form, without =value) modifies the configuration-level ActionEnvironment stored on BuildConfigurationValue. But it only manifests at action execution time in actions where use_default_shell_env=True.

From SpawnAction.java (lines 595-626), createActionEnvironment():

So --action_env=PATH doesn’t inject PATH into cc_library, py_binary, or any other action that uses the default use_default_shell_env=False. It only affects actions that already opted into the shell environment.

Claim: “–action_env=PATH pollutes cache keys for every action”

Misleading.

From ActionEnvironment.java (lines 134-137), addTo(Fingerprint): the variable name (not value) goes into the action key fingerprint for actions that inherit it. The value is tracked by Skyframe via CLIENT_ENVIRONMENT_VARIABLE — changing your actual $PATH between builds correctly invalidates affected actions.

But again — “affected actions” means actions with use_default_shell_env=True. The fingerprint change doesn’t touch actions that never see the variable.

What --action_env=PATH does do: it changes the configuration hash, which can trigger reanalysis when added or removed. That’s a one-time cost, not an ongoing cache pollution.

Claim: “use_default_shell_env=True only leaks PATH”

Wrong.

I said this scopes the leak to “just PATH for one action.” That’s not accurate either.

From BazelRuleClassProvider.java (lines 137-205), SHELL_ACTION_ENV on Linux passes: PATH, LD_LIBRARY_PATH, and LC_CTYPE. Plus anything added by --action_env. On macOS, also SDKROOT.

It’s not the full host environment — just 3-4 variables from the shell action env. But it’s more than “just PATH.”

Claim: “Our approach scopes the leak, Ryan’s is global”

Technically true but practically meaningless.

Both PRs set use_default_shell_env=True on the sdist_build action. Both read from the same SHELL_ACTION_ENV pipeline. The “leak” is identical in both.

On Linux, PATH is already in SHELL_ACTION_ENV by default (BazelRuleClassProvider.java, line 176). Ryan’s --action_env=PATH is a no-op for PATH — it’s already there.

Ryan’s --action_env=CC and --action_env=CXX do add new variables, but they only manifest in actions with use_default_shell_env=True — which is the sdist_build action. Same scope.

The actual difference between the PRs

Aspect Ryan’s #21967 My #22578
Core mechanism use_default_shell_env=True in patch use_default_shell_env=True in patch
Extra .bazelrc flags --action_env=PATH/CC/CXX/SDKROOT None
CC toolchain registration None use_cc_toolchain(mandatory=False) (no-op today)
LLVM toolchain None toolchains_llvm registered (unused by sdist builds)
Smoke test None test_fasttext_smoke

The core fix is the same. I added toolchain scaffolding and a test. Ryan added explicit env forwarding flags that are mostly redundant on Linux. Neither approach is meaningfully “more hermetic” than the other — I was wrong to claim otherwise.

The root cause, verified

The reason hermetic CC toolchains don’t work with sdist_build is verified and straightforward:

  1. build_helper.py in aspect_rules_py changes CWD to a temp directory before running python -m build
  2. cc_wrapper.sh from toolchains_llvm resolves the clang binary via a relative path: external/toolchains_llvm++llvm+llvm_toolchain_llvm/
  3. That relative path is relative to Bazel’s execroot. From a temp directory, it resolves to nothing.
  4. fasttext’s setup.py has a has_flag() function that catches CompileError silently — so the failure mode is “C++17 not supported” rather than “compiler not found”

Verified by reading build_helper.py, cc_wrapper.sh, and the CI failure logs.

Note: rules_python doesn’t have this problem because pip.parse() downloads pre-compiled wheels. The sdist issue is specific to aspect_rules_py’s uv extension, which reads uv.lock and builds sdists when no wheel is available.

The side fixes

The five attempts were the main arc. CI had opinions about other things too.

Buildifier formatting. MODULE.bazel needed alphabetical argument ordering. Every time I touched it, the lint check caught me.

Stale lockfile. Adding toolchains_llvm v1.6.0 pulled in a transitive dependency update (tar.bzl 0.5.5 to 0.6.0). The lockfile diff was thousands of lines for one MODULE.bazel change.

macOS compatibility. No macOS SDK headers for hermetic LLVM. Added target_compatible_with = ["@platforms//os:linux"] to the test target.

build_tests_only. bazel test //... --test_tag_filters=torch still builds all targets before filtering which to run. Added --build_tests_only so it only builds what matches.

What I actually learned

Read the source code before making claims. I wrote a PR description arguing about Bazel environment semantics based on documentation and intuition. The intuition was wrong. The source code (SpawnAction.java, ActionEnvironment.java, BazelRuleClassProvider.java) told a different story. If your claims are about implementation behavior, read the implementation.

The CWD problem is real and fundamental. build_helper.py changes CWD to a temp dir. Everything in toolchains_llvm uses execroot-relative paths. These are structurally incompatible. The 8-line fix (resolve paths before CWD change) would work but requires patching build_helper.py.

use_default_shell_env=True is not “the full host environment.” It passes SHELL_ACTION_ENV — on Linux, that’s PATH, LD_LIBRARY_PATH, and LC_CTYPE. Not /etc/environment, not your .bashrc exports, not arbitrary host state. It’s a controlled, small set of variables. Still not hermetic, but less leaky than I initially described.

--action_env is scoped, not global. It only manifests in actions that use use_default_shell_env=True. It doesn’t inject variables into every action in the build graph. I said it did. I was wrong.

Being wrong in public is fine if you correct it. I made claims in a PR description that didn’t hold up. That’s embarrassing but fixable. What would be worse is leaving the wrong mental model in place — both for myself and for anyone who reads the PR later.

References

Bazel source files cited

External references