Bazel Migration: 62% Blocked by torch

TL;DR: Without AI/GPU dependencies, maintaining a Python Bazel monorepo is genuinely pleasant. The moment torch, CUDA, and flash-attn enter the picture, 62% of your packages become unmigrateable. Here’s the dependency wall, the single config package bottlenecking everything, and the baby-step plan I’m executing starting today.

This is the companion post to When GPU Showed Up in Our Python Bazel Monorepo. That post was observational — “here’s what happens when GPU arrives.” This one is actionable — “here’s the wall, and here’s the plan to punch through it.”

The Scoreboard

We have 95 workspace packages. Here’s where they stand:

Category	Count	%
Already configured	8	8.4%
Ready (no blockers)	23	24.2%
Blocked by deps	59	62.1%
Blocked by C++ code	4	4.2%
Skipped	1	1.1%

Eight done. Twenty-three ready to go. And then a wall.

Fifty-nine packages sitting in dependency limbo — blocked by native/GPU deps that Bazel’s rules_python can’t resolve. The maddening part? Most of those 59 packages don’t even use GPU. They just happen to import something that imports something that eventually reaches torch.

They’re innocent bystanders in a dependency graph crime.

The Gateway Blocker

Here’s the cascade:

                    torch >= 2.7.0     numpy == 1.*
                         \              /
                          \            /
                     +------------------+
                     |   mai_config     |  <-- THE GATEWAY BLOCKER
                     | (torch + numpy)  |      blocks 22 direct, ~50 transitive
                     +--------+---------+
                              |
              +---------------+---------------+
              |               |               |
         mai_cluster    mai_preprocessors   mai_numerics
              |               |
         mai_distributed   mai_logger
              |               |
              +-------+-------+
                      |
          mai_layers -> mai_trainer -> mai_multimodal -> simplertransformer

Let me say this again, slowly:

mai_config is a configuration package. It manages settings. Feature flags. Environment detection. Stuff like “where do the log files go” and “which model registry are we hitting.”

It depends on torch>=2.7.0.

A 68MB GPU framework. For a config package.

And because everything reads config, everything transitively depends on torch. This one package blocks ~22 direct dependents and ~50 packages transitively. It’s the single biggest bottleneck in the entire migration.

There are two more chokepoints worth calling out:

conversation depends on both mai_config and mai_preprocessors, cascading the blockage to 30+ more packages
lib/bus brings in C++ via Cython + numpy build deps, blocking lib/oai_api and 15+ packages downstream

But mai_config is the gateway. Fix that, and the flood gates open.

Deep Dive: Each Blocker

Not all blockers are created equal. Some are paper tigers. Some are real walls. Let’s sort them out.

numpy: Not Actually a Blocker

Plot twist: numpy is fine.

@pypi//numpy already works in our Bazel setup. lib/mai_nano uses it today — prebuilt wheels resolve, tests pass, zero drama. numpy only looks like a blocker because it shows up in packages that also need torch. Peel away torch and numpy would just work.

Verdict: Not a blocker. Move on.

torch: The Real One

Torch is genuinely hard, and it’s hard in two completely different ways depending on your platform:

macOS (dev machines):

torch comes from PyPI as a normal wheel (68MB, arm64)
@pypi//torch should resolve it — it’s just a fat wheel
We haven’t tested this yet. But the mechanism exists.

Linux (CI/training):

torch is system-installed inside Docker images
It’s NOT in uv.lock — the root pyproject.toml says:

"torch; sys_platform == 'darwin'"

Translation: uv only resolves torch on Mac. On Linux, the Docker image provides it.
Bazel has no built-in concept of “torch is already installed, don’t try to resolve it”

So on Mac, torch might work through normal pip resolution. On Linux, we need a fundamentally different mechanism — some kind of system-provided package override.

Two platforms, two completely different problems. Welcome to GPU land.

flash-attn: The Ghost

This one gets creative:

# Root pyproject.toml -- uv override-dependencies
"flash-attn; sys_platform == 'never'"
"triton; sys_platform == 'never'"
"ucxx-cu12; sys_platform == 'never'"

sys_platform == 'never' – a platform that doesn’t exist. This tells uv: “pretend this package doesn’t exist during resolution.” It’s always system-installed (via pip in Docker), never touched by the package manager.

Clever hack. The kind of hack you write at 2 AM and then find surprisingly hard to replace with anything better.

But Bazel doesn’t speak this dialect. Packages that import flash_attn have a dependency Bazel can’t see, can’t resolve, and can’t verify. It’s a dangling reference in the build graph — a ghost in the dependency tree.

CUDA Packages (mai_kernels, simplertransformer): The Deep End

Four packages contain actual C++ and CUDA code, built via scikit-build-core + CMake. These are the “Blocked by C++ Code” category.

rules_python can’t build CUDA kernels. Full stop. You’d need rules_cuda (which barely exists as a maintained project) or a pre-built wheel strategy where you build the wheels externally and consume them in Bazel.

These are the hardest packages to migrate. We’re explicitly deferring them.

The Baby-Step Plan

Here’s what we can actually do, ordered by impact-to-effort ratio.

Step 1: Harvest the 23 Ready Packages

Impact: HIGH | Effort: LOW | When: Today

Twenty-three packages are sitting there with zero native dependency blockers. Pure Python, deps resolve fine via @pypi//, and the only reason they’re not configured is nobody’s run gazelle on them yet.

Just… do it. Go from 8 to 31 configured packages. That’s nearly 4x coverage. Each one is a gazelle run, maybe a small BUILD fixup, and a commit.

Not glamorous. But it moves the scoreboard from 8.4% to 32.6%. That number matters when you’re building momentum (or trying to convince yourself this migration is real).

Step 2: Audit mai_config’s torch Usage

Impact: VERY HIGH | Effort: MEDIUM | When: This week

This is the highest-leverage investigation in the entire migration. One question:

Does mai_config actually import torch at module load time? Or is it a lazy import hiding in some utility function?

If torch is only used in one function that checks GPU availability — and we can make it a lazy import behind TYPE_CHECKING or a try/except — we unblock ~50 transitive packages in a single PR.

A config package shouldn’t need to import torch at the top level. It probably doesn’t. But someone has to go look. And “someone” is me.

Step 3: Smoke-Test @pypi//torch on macOS

Impact: MEDIUM | Effort: LOW | When: This week

The simplest possible validation:

# test_torch_import.py
import torch
print(f"torch version: {torch.__version__}")
print(f"CUDA available: {torch.cuda.is_available()}")

# BUILD.bazel
py_test(
    name = "test_torch_import",
    srcs = ["test_torch_import.py"],
    deps = ["@pypi//torch"],
)

Run it. If torch resolves and imports on macOS through @pypi//, we’ve validated the Mac-first migration strategy. If it blows up, we know exactly where the gap is.

15 minutes. One answer. Gates everything downstream.

Step 4: Migrate tokenization and lib/sydney

Impact: MEDIUM | Effort: LOW | When: Next sprint

Both are pure Python, no direct torch dependency, already on the “ready” list. Good candidates to demonstrate that migration works for real code — not just leaf utilities nobody touches.

Step 5: Plan System-Torch Override for Linux

Impact: HIGH | Effort: MEDIUM-HIGH | When: Next month

The big architectural question: how do we tell Bazel “torch is provided by the Docker environment, don’t resolve it”?

aspect_rules_py’s uv extension supports overrides in MODULE.bazel. We need a spike to figure out the exact mechanism — mapping the torch requirement to a system-provided path.

This is what determines whether Linux CI works at all. Not urgent (Mac-first gets us far), but needs a spike soon.

Step 6: Treat lib/bus as a Pre-Built Wheel

Impact: MEDIUM | Effort: MEDIUM-HIGH | When: Future

lib/bus ships .so files via Cython + numpy build deps. Rather than teaching Bazel to build Cython (pain), we could package it as a pre-built wheel and consume it as an external dependency.

This “build externally, consume in Bazel” pattern might become the template for all our C++/CUDA packages. Worth getting right.

What We’re Explicitly Not Doing

CUDA kernel compilation in Bazel – enormous effort, unclear payoff. Docker handles this fine today.
Full Linux GPU support in Bazel – out of scope until the system-override mechanism is proven on one package.
mai_kernels and simplertransformer – need rules_cuda or pre-built wheels. Not this quarter.
flash-attn, triton in Bazel – need a “system-provided” override mechanism. Deferred.

Deferring is not ignoring. It’s acknowledging that some problems need the earlier steps solved first.

The End State

Pure Python deps    -> @pypi// from uv.lock           done, working today
numpy, scipy        -> prebuilt wheels                 done, working today
torch (macOS)       -> @pypi//torch wheels             step 3 validates this
torch (Linux)       -> system override in MODULE.bazel step 5 designs this
CUDA packages       -> pre-built wheels or custom      deferred
flash-attn, triton  -> system-provided override        deferred

The strategy is concentric circles: migrate everything you can today, then push the boundary outward. Pure Python → numpy-tier → torch-dependent → CUDA-native.

The Uncomfortable Truth

Without AI/GPU dependencies, maintaining a Python Bazel monorepo is… easy. Actually pleasant. rules_python + gazelle + uv.lock gives you hermetic builds, fast caching, and one-command test runs. It works the way the documentation says it does.

The thing that makes this migration hard isn’t Bazel. It’s that the Python AI/ML ecosystem was built around a different assumption: pip install inside a Docker container with CUDA pre-installed. System-level deps, platform-specific wheels, GPU driver version coupling – none of this was designed for hermetic build systems.

We’re not fighting Bazel. We’re fighting the assumption, baked into every GPU library, that “the environment will provide.”

And 62% of our packages are stuck behind that assumption.

But step 1 is just running gazelle on 23 packages. That, I can do this afternoon.

Field notes from migrating a 95-package Python monorepo to Bazel. The dependency graph, blocker percentages, and sys_platform == 'never' hack are all from a real codebase. The frustration about a config package depending on torch is also real.

Previously: When GPU Showed Up in Our Python Bazel Monorepo – the observational companion piece about what happens when GPU workloads arrive in your clean Bazel monorepo.