Python Packaging: Native Code in a Monorepo

Migrating a Python monorepo to Bazel is straightforward until the first ImportError: No module named '..._schema' shows up. Then you’re in native code territory, and the rules change.

This post started as a TIL about what cc means in Bazel’s naming. It turned into the full story of how native compilation works (and breaks) in a Python monorepo migration — C extensions, C++, Cython, two fundamentally different build paths, and a plot twist where the scariest blocker in our monorepo turned out to be four .pyx files pretending to be a C++ library.

The naming fossil

cc in Bazel — cc_toolchain, rules_cc, cc_library — stands for C Compiler.

Not “C/C++.” Not “compilation chain.” Just “C Compiler.” The name of the original Unix command: /usr/bin/cc.

cc was the C compiler on Unix. The literal command you typed. When C++ showed up, the tooling grew to handle both languages, but nobody renamed the command. Same thing happened with GCC — started life as “GNU C Compiler,” quietly became “GNU Compiler Collection” as it absorbed C++, Fortran, and everything else. The binary is still called gcc.

Bazel inherited this convention wholesale. cc_library compiles C and C++. cc_toolchain configures the compiler, linker, and flags for both. The name says C; the reality is C/C++.

The Bazel cc stack

Three layers, bottom to top:

The rules_common_cc name decomposes neatly: rules (Bazel rule definitions) + common (shared base used by multiple language rulesets) + cc (the C/C++ portion). It’s not “common C rules” — it’s “the common layer, for cc.” The plumbing, not the porcelain.

Registered toolchain vs. cc_library targets

This is the conceptual split that most people miss when they first encounter the CC stack.

You can register a CC toolchain without having a single cc_library target in your repo. All it means is: “if anything needs to compile C/C++, here’s the compiler.” It’s the stove sitting idle in the kitchen. Nobody’s cooking yet.

The confusion comes from the fact that you might need the toolchain for two completely different reasons — and those reasons demand very different amounts of effort. Which brings us to the two build paths.

Three flavors of native code in Python

Before the build paths, a quick taxonomy. There are three ways fast native code ends up in a Python project, and they’re not the same thing:

C extension (e.g., fasttext). You (or more likely someone upstream) write C and a setup.py that calls the compiler. pip install invokes gcc behind the scenes. You’re dealing with Python’s C API directly — PyObject*, reference counting, the whole adventure. The result is a .so that Python can import.

C++ (standalone libraries). Pure C++ code that gets wrapped for Python consumption — maybe via pybind11, maybe via a hand-rolled wrapper. The C++ exists independently; Python just calls into it. Needs cc_library / cc_binary rules in Bazel.

Cython (e.g., lib/bus). The bridge language. You write Python-like .pyx files, Cython generates .c (or .cpp), and gcc compiles that to .so. It’s a code generator that produces C extensions for you, so you don’t have to write the PyObject* nightmare by hand.

The Bazel difficulty column is the key. Pure Python: trivial. C extensions via pip: medium (you just need the compiler available). Anything Bazel compiles natively: hard. That distinction maps directly onto the two build paths.

The two build paths

This is the mental model that makes everything else click.

Path 1: "pip install" path (sdist)        Path 2: "native Bazel" path
─────────────────────────────             ──────────────────────────
Bazel says: "pip, build this wheel"       Bazel says: "I'll compile this myself"
pip calls setup.py → gcc → .so           Bazel calls cc_library / cython rule → .so
Bazel treats the result as a black box    Bazel tracks every source file and dep

Path 1 is what happens with @pypi// dependencies. Bazel delegates to pip, pip runs setup.py, the compiler runs, a wheel pops out, Bazel consumes it without knowing or caring what happened inside. The CC toolchain registration is still needed — pip needs to find the compiler — but Bazel itself never invokes cc_library. It’s just making sure the stove exists so pip can use it.

Path 2 is what happens with //lib/... dependencies — code that lives in your monorepo and has BUILD files. Bazel compiles it directly, tracking every source file, header, and dependency. This is the full cc_library / cc_binary / Cython-rules path. Bazel is doing the cooking.

The practical split:

@pypi// deps → Path 1. //lib/... deps → Path 2.** They want different things from the CC toolchain.

The reason this matters: when someone says “we need a CC toolchain for our Python monorepo,” the follow-up question is which path? Path 1 is a config change in MODULE.bazel and maybe a use_default_shell_env=True flag. Path 2 is a build system architecture project.

What happens when things are missing

The failure modes are different depending on which piece is absent:

The first error — No matching toolchains found — is the one that sends people down rabbit holes. It reads like a Bazel configuration problem. And it is. But knowing whether you need the toolchain for Path 1 (pip needs a compiler) vs. Path 2 (Bazel needs to compile natively) determines whether the fix is a one-line config change or a multi-week project.

Three states of a package

In a migration that’s still in progress, packages exist in one of three states:

Not Bazelized. Listed in gazelle:exclude, Bazel doesn’t know the package exists. It’s a poison pill — anything that depends on it can’t migrate either, and that propagates transitively through the dep graph.

Partially Bazelized. The Python parts have BUILD files. Gazelle generated targets for the .py files. Type checking works because Bazel can see the .pyi stubs. But the native extensions — the .so files that Cython or C code produces — are missing. At runtime:

ImportError: No module named 'oai.multimodal_token.token_schema.token_schema'

The Python files exist. The compiled extension that they expect to import doesn’t. Bazel built the parts it knew how to build and silently left out the rest.

Fully Bazelized (via pip). @pypi//fasttext — pip handles the entire build, Bazel treats the wheel as a black box. Works as long as pip can find a compiler. No BUILD files needed for the package internals.

The lib/bus plot twist

Here’s where the story gets good.

lib/bus is the package that transitively blocks ~28 other packages in our monorepo. It shows up in every migration status report as the primary “poison pill” — the thing preventing the next wave of packages from migrating. It was labeled “C++/Cython” in our tracking, and everyone (including me) assumed it meant: C++ source code, Cython bindings, the whole native compilation nightmare. Full Path 2 territory. A multi-week effort to get Bazel to build it natively.

So I went to look at the actual source code.

Zero C++ source files. Zero. No .cpp, no .cc, no .cxx. Nothing.

Zero CUDA files. Despite being in an ML monorepo, not a single .cu.

The “C++” label came from one line in setup.py:

language="c++"

That’s Cython’s option for which compiler backend to use when generating C code from .pyx files. It doesn’t mean there’s C++ source code. It means Cython’s code generator outputs .cpp instead of .c, so it can use C++ features in the generated code. The source is still .pyx.

The actual inventory of lib/bus:

• 420 Python files — just… Python
• 4 .pyx files in oai/multimodal_token/ — implementing Token dataclasses and batch operations
• 0 C++ files
• 0 CUDA files

Four Cython files. That’s the thing blocking 28 packages.

And here’s the kicker: those four files implement Token dataclasses and batch operations. They’re a performance optimization — Cython makes attribute access and iteration faster by compiling to native code. But they’re not doing FFI. They’re not calling into a C library. They’re not wrapping CUDA kernels. They’re doing what @dataclass(slots=True) does, just faster.

The pragmatic option: rewrite those 4 files as pure Python. @dataclass(slots=True) + IntEnum + maybe some __slots__ tuning. The native code problem doesn’t get solved — it gets deleted.

Whether the performance hit is acceptable is a real question. But it’s a question about priorities, not about Bazel build rules. And it’s a very different conversation than “we need to implement Cython rules in Bazel and register a CC toolchain and configure cross-compilation for our CI runners.”

Sometimes the biggest blocker isn’t what you think it is. Sometimes it’s four files cosplaying as a C++ library.

The full ruleset map

Now that we’ve covered the native code landscape, here’s the complete inventory of what a Python-heavy ML monorepo needs in Bazel — what we have, and what’s missing.

What we already have

These live in MODULE.bazel on main today:

This covers pure Python. Which is most of the monorepo — roughly 95% of the code by file count. If your package has no native code and no torch dependency, these rulesets are all you need, and migration takes about ten minutes per package.

What’s missing

Each missing ruleset corresponds to a different class of native dependency. They’re not equally important, and — critically — they map to different build paths.

1. rules_cc + CC toolchain (e.g., toolchains_llvm)

C/C++ compilation rules plus a registered compiler. Needed for both build paths: Path 1 (pip needs a compiler for sdist packages like fasttext) and Path 2 (Bazel compiles C/C++ targets natively).

We don’t have this on main yet. There’s a PR that adds toolchains_llvm + rules_cc, scoped only to sdist builds. The twist: the biggest “C++” blocker in our monorepo is actually pure Cython. The CC toolchain alone doesn’t unblock it.

2. Cython rules (no good standard option)

Rules to handle the .pyx → .c/.cpp → .so compilation pipeline. This is Path 2 territory — Bazel compiling natively.

This is the actual bottleneck. The community rules_cython is poorly maintained. Your options are: custom genrule + cc_library pipeline (works but ugly), or avoid the problem entirely by rewriting the Cython as pure Python.

3. rules_cuda

CUDA compilation rules for .cu files, nvcc integration. Path 2, exclusively — there’s no pip equivalent for custom CUDA kernels.

Not needed for the Cython blocker (zero CUDA files), but needed for packages like mai_kernels. CUDA rules only work on machines with CUDA installed, so macOS and CPU-only CI need target_compatible_with exclusions — same pattern we already use for torch tests.

4. rules_foreign_cc

Build non-Bazel C/C++ projects (CMake, make, autoconf) from within Bazel. You’d need this if you vendored a C library that uses CMake.

We don’t need this today. “Maybe someday” territory.

5. Protobuf / gRPC rules (rules_proto, etc.)

Not needed. Proto handling happens at the Python level via pip packages. If your monorepo generates .proto stubs at build time, you’d need these. We don’t.

The migration sequence

The naive mental model is: “Python monorepo with native deps → need CC toolchain → hard.” The reality is a sequence of five phases, each unlocking a different set of packages, each requiring different tooling:

Phase 1 is where you get 60% of the monorepo migrated with tooling that already exists. Phase 2 is where we just landed — system torch in Docker. Phase 3 is in-progress. Phases 4 and 5 are the long tail.

The pattern is clear: complexity per package goes up as you move right, while package count goes down. The build path column tells you why — Path 1 packages are a config problem, Path 2 packages are a build system architecture problem.

The punchline: rules_python handles 95% of the work. The CC toolchain unlocks a few pip packages. The real blockers are Cython and CUDA rules — and for Cython specifically, the pragmatic answer might be “don’t use Cython” rather than “add Cython build rules to Bazel.”

The actual lesson

The migration sequence matters more than any individual ruleset. Pure Python first — biggest bang for buck, lowest effort. Then system torch. Then C extensions via pip. Then Cython and CUDA last, where the cost per package is highest and you have time to decide whether “rewrite the native code” is a better answer than “teach Bazel to build it.”

And before you scope out a multi-week project to add native compilation support to your Bazel setup — go read the source code. Count the actual .cpp files. Count the actual .pyx files. Check whether the “C++ library” blocking your migration is really a C++ library, or four Cython files that could be a dataclass.

Sometimes the hardest part of a build system migration isn’t the build system. It’s knowing what the words mean.