Bazel Migration: Common Blockers and the Roadmap Out

We have 106 packages in our Python monorepo. 28 are on Bazel. That’s 26%. The other 78 are stuck behind a wall of native code – Cython extensions, C++ libraries, CUDA kernels – and no amount of Python-level cleverness will fix it.

This post is the strategy briefing. What’s actually blocking us, what it would take to unblock each thing, and where the ROI is highest. If you’re an engineering lead staring at a Bazel migration that plateaued at ~25%, this is the map.

The four dimensions

Before we get to the blockers, you need a mental model. There are four orthogonal concepts that all interact when you’re building native code with Bazel, and every blocker we’ll talk about is a problem in one or more of them.

1. Rules – the “what you build”

Bazel rules are build instructions for a specific kind of artifact. The ones that matter here:

• rules_cc – plain C/C++ compilation. Source files in, .so/.a out.
• rules_cython – .pyx files compiled through Cython to C, then to shared libraries. Python C extensions.
• rules_cuda – .cu files compiled through nvcc into GPU kernels.

The critical dependency: rules_cython and rules_cuda both depend on rules_cc under the hood. Cython generates C code that needs a C compiler. CUDA needs a host compiler for the non-GPU parts. You always need working C++ first. Always.

Think of it as a stack:

rules_cuda    rules_cython
     \            /
      \          /
       rules_cc        ← foundation

If your C++ toolchain doesn’t work, nothing above it works either.

2. Toolchain – the “what builds it”

A toolchain is the set of programs that do the actual compiling and linking. Three that matter:

• C++ toolchain – gcc or clang, a linker, system headers. The bread and butter.
• CUDA toolchain – nvcc (NVIDIA’s compiler), the CUDA SDK, GPU-specific headers.
• Python toolchain – the python3 binary, site-packages, the interpreter itself.

Here’s the analogy: rules are recipes. Toolchains are kitchens. Same recipe, different kitchen – you can compile the same rules_cc target with GCC on Linux or Clang on macOS. The rule says what to build. The toolchain says how.

Bazel’s toolchain resolution is a matchmaking service. A build target says “I need to compile C++ for linux-x86_64.” Bazel scans registered toolchains and picks one that satisfies the constraint. If nothing matches, the build fails. If nobody registered a CUDA toolchain, rules_cuda targets have no kitchen to cook in.

3. Platform – the “where”

Platforms define the constraints of the build and execution environment. Two mechanisms:

target_compatible_with – “skip me if these constraints aren’t met.” A CUDA test tagged with @platforms//os:linux and @platforms//gpu:nvidia will silently skip on macOS instead of failing. This is how you keep your bazel test //... green on machines that can’t possibly run GPU code.

select() – “use different inputs depending on platform.” Different source files for Linux vs macOS. Different compiler flags for GPU vs CPU. Different dependencies entirely.

The platform hierarchy creates a natural capability ladder:

Each step up the ladder adds capabilities. macOS can build Python and C++, but not Cython (we don’t ship the right numpy headers) and definitely not CUDA. Linux without a GPU can compile CUDA code but not run it – so tests skip. Full GPU nodes can do everything.

This matters because your CI matrix has to match. You don’t want a Cython build breaking your macOS developer experience, and you don’t want CUDA tests blocking merges when no GPU runner is available.

4. Package vs library – the “what you get”

This one trips people up because “package” means two different things.

Bazel “package” – a directory with a BUILD file. It’s a namespace for targets. //lib/bus:bus is a target in the Bazel package lib/bus.

Python “package” – an installable thing with a pyproject.toml. It goes on PyPI (or your internal registry). pip install mai_config.

A pure Python package is simple: your BUILD file has py_library targets and maybe py_test. One concept, one rule.

A Python package with C extensions needs both worlds:

cc_library    → compiled C/C++ code (.o, .a)
py_extension  → the .so bridge that Python can import
py_library    → pure Python source files

py_extension is the bridge – it takes compiled native code and wraps it so Python’s import machinery can load it. When you import bus._native, Python is loading a .so file that was compiled from C/Cython, not reading a .py file. The BUILD file has to express this entire chain: C source -> compiled object -> Python-importable extension -> Python library that imports it.

This is why native packages are hard to migrate. A pure Python package migration is mostly gazelle auto-generating a BUILD file. A native package migration is hand-writing build rules that express compilation, linking, platform constraints, and the bridge between two languages.

The actual blockers

Here’s the scoreboard, right now:

That 63% is the wall. Let’s break it down by what’s actually in the way.

Notice how the four dimensions show up in every row. lib/bus needs Cython rules, a C++ toolchain with numpy headers, and only works on the Linux platform. mai_kernels needs CUDA rules, a CUDA toolchain, and a GPU platform for testing. The framework isn’t just conceptual – it tells you exactly what you need to solve for each blocker.

The poison pills

Two packages deserve special attention because their blast radius is disproportionate.

lib/bus is the biggest single blocker. It’s a Cython package – .pyx files that compile to C extensions. By itself, that’s just one hard migration. The problem is the dependency chain: lib/bus -> lib/oai_api -> common_utils -> everything. lib/bus infects roughly 28 downstream packages through transitive dependencies. Many of those downstream packages have never heard of Cython. They just import something that imports something that eventually reaches lib/bus. Innocent bystanders.

lib/excel_evaluation is the hidden kingmaker. It’s the sole blocker for mai_evaluator, which is the sole blocker for rocket, which gates the entire evaluation, training, and RL stack. One C++ package, holding hostage the entire training pipeline. Nobody talks about it because lib/bus gets all the attention, but lib/excel_evaluation unblocks just as many packages.

The fake blocker

mai_multimodal and speaker_diarization have a circular dependency – according to our migration tooling. In reality, mai_multimodal only depends on speaker_diarization through a benchmark script. Not a real import. Not a runtime dependency. The migration scanner is too conservative – it sees import speaker_diarization in any .py file under the package and flags it as a hard dependency.

Fixing this is a tooling change, not a code change. Either exclude benchmark files from the scanner, or restructure the benchmark into its own package. This is a couple hours of work that unblocks 9 packages. Free money.

The roadmap

Five phases, ordered by effort-to-impact ratio.

Phase 1: Free money (+19 packages, 26% -> 44%)

What: Migrate 3 ready packages + fix the fake mai_multimodal cycle.

Effort: Days, not weeks.

Why it’s free: These packages have zero native code. They’re pure Python, all deps already on Bazel. The only reason they’re not migrated is that nobody’s gotten to them yet. The cycle fix is a tooling patch. This is 19 packages for a few days of mechanical work.

Dimensions involved: None – this is all Python-level. No new rules, no new toolchains, no platform constraints.

Phase 2: The Cython wall (+12 packages, 44% -> 56%)

What: Migrate lib/bus.

Effort: Weeks. This is real engineering.

What’s needed:

• Rules: rules_cython for .pyx -> .c -> .so compilation, backed by rules_cc
• Toolchain: C++ compiler with numpy headers available (for Cython’s generated C code that #include <numpy/arrayobject.h>)
• Platform: Linux-only. Cython compilation with numpy headers doesn’t work on macOS without extra setup, and there’s no reason to invest in that – our CI and production are Linux.

The hard part: Cython’s build process is finicky. The generated C code depends on the exact Cython version, numpy version, and Python version. Getting the header paths right in a sandboxed Bazel build – where nothing is on the system $PATH – requires careful toolchain configuration. There’s no off-the-shelf rules_cython that handles our setup. We’d likely need to write custom rules or heavily adapt an existing one.

Why it’s worth it: lib/bus is the biggest poison pill. Fixing it breaks the lib/bus -> lib/oai_api -> common_utils -> world chain and unblocks 12 packages in one shot. Some of those 12 are themselves transitive blockers, so the real cascade is even larger.

Phase 3: The hidden kingmaker (+12 packages, 56% -> 67%)

What: Migrate lib/excel_evaluation.

Effort: Unknown until investigated, but likely weeks.

What’s needed:

• Rules: Probably rules_cc, but we haven’t fully investigated the build yet
• Toolchain: C++ compiler, potentially with specific library dependencies
• Platform: Unknown – needs investigation

The hard part: We haven’t done a deep dive on lib/excel_evaluation’s native code yet. It could be straightforward C++ with rules_cc, or it could have exotic dependencies. The investigation itself is the first task.

Why it’s worth it: Same math as lib/bus. lib/excel_evaluation -> mai_evaluator -> rocket -> the entire eval/training/RL stack. Twelve packages behind one door. And unlike lib/bus, this one might turn out to be simpler – it’s C++, not Cython, so there’s no .pyx -> .c generation step to worry about.

Phase 4: CUDA territory (+19 packages, 67% -> 85%)

What: Migrate mai_kernels and simplertransformer.

Effort: Multiple weeks to months. This is the hard stuff.

What’s needed:

• Rules: rules_cuda + rules_cc. CUDA compilation involves splitting code into host (C++) and device (GPU kernel) parts, compiling them with different compilers, and linking them together.
• Toolchain: CUDA SDK (nvcc), a compatible C++ compiler (CUDA is picky about which GCC versions it works with), GPU-specific headers. For simplertransformer, also AMD ROCm support – a second GPU vendor’s entire toolchain.
• Platform: Full GPU nodes for testing. Compile-only possible on Linux without GPU, but you can’t verify correctness without actual hardware.

The hard part: Everything. CUDA in Bazel is cutting-edge. rules_cuda exists but is immature compared to rules_cc. CUDA’s two-phase compilation (host + device) doesn’t map cleanly to Bazel’s action model. nvcc has its own preprocessor, its own linker requirements, its own version compatibility matrix with the host compiler. And simplertransformer adds AMD ROCm to the mix – an entirely separate GPU ecosystem with different compilers (hipcc), different libraries, and different build flags.

The CI infrastructure is also a concern. GPU runners are expensive and scarce. You need a CI matrix that compiles CUDA code on regular Linux nodes (cheap) but only runs GPU tests on GPU nodes (expensive). The target_compatible_with mechanism handles the build-level gating, but the CI workflow routing is its own problem.

Why it’s still worth it: 19 packages is almost 20% of the monorepo. And these are the heavy-hitters – the actual model training and inference code. Getting them on Bazel means your core ML pipeline gets Bazel’s caching and hermeticity.

Phase 5: Cleanup (+2 packages, 85% -> 87%)

What: Stragglers. Packages with one-off blockers or minor issues.

Effort: Variable. Each one is its own puzzle.

The remaining ~13% that never makes it is a mix of packages that are deprecated, in the process of being removed, or have such exotic build requirements that the migration cost exceeds the benefit.

The math

Phase 1 is the obvious next move. 19 packages for days of work is the best ROI you’ll ever see in a migration. Phases 2 and 3 are where the real investment starts – and where understanding the four dimensions becomes essential, because you’re making decisions about which rules to adopt, which toolchains to configure, and which platforms to support.

Phase 4 is the long game. It’s the most engineering-intensive, but it’s also where the highest-value packages live. If your monorepo’s core product is ML training and inference, leaving that code off Bazel means your most important builds don’t get Bazel’s benefits.

What this means for planning

Three things to take away:

The migration isn’t stuck – it’s staged. The 26% plateau isn’t a failure of the approach. It’s the natural boundary between “pure Python packages that Gazelle auto-migrates” and “native code packages that need manual build engineering.” Every large Python monorepo with native dependencies hits this wall.

The four dimensions are your diagnostic tool. When someone asks “why can’t we migrate package X?”, the answer is always a combination of: which rules does it need, do we have the toolchain, and which platforms does it target. If you can answer those three questions, you know the migration path. If you can’t, that’s the investigation to run first.

The ROI curve is front-loaded. Phase 1 delivers 19 packages for days of effort. Phase 4 delivers 19 packages for weeks-to-months. Start at the top. The easy wins aren’t just good for morale – they reduce the surface area of the problem, making the hard phases more tractable.

The native code wall is real, but it’s not infinite. It’s four packages – lib/bus, lib/excel_evaluation, mai_kernels, simplertransformer – that between them block 63% of the monorepo. Fix them in order of difficulty, and the migration curve starts moving again.