TL;DR: Before writing a single line of code to make PyTorch work in Bazel, we needed to know: is it even worth it? We ran a wave-by-wave cascade analysis on our dependency graph and found that fixing torch would unblock 30 packages — taking us from 24% to 53% Bazel coverage. An earlier approach (lazy imports) would have unblocked exactly 1. The analysis took a few hours. It saved us weeks of wasted effort.
This is a follow-up to the Bazel migration battle plan and the dependency resolution deep dive. Those posts covered the wall and the machinery. This one covers the math — how we quantified the impact before committing to a fix.
The Question That Started It
We had three options for making torch work in Bazel. Each one had different engineering cost, different risk, and different blast radius. Before picking one, we needed to answer a deceptively simple question:
If torch becomes available in Bazel tomorrow, how many packages actually get unblocked?
The answer isn’t obvious. Unblocking torch doesn’t just unblock the 20 packages that directly import torch. It potentially unblocks everything downstream of those packages too — a cascade through the dependency graph. But only if all of a downstream package’s blockers are resolved, not just one.
And some packages are permanently stuck. C++ code, Cython extensions, packages excluded for other reasons — these are poison pills. They block everything below them no matter what you fix upstream.
So you can’t just count direct dependents. You need to simulate the cascade.
The Source of Truth
Everything starts with bazel/MIGRATION_STATUS.md — a machine-generated file that categorizes all 101 packages in the monorepo. It’s produced by bazel/tools/find_bazel_ready_packages.py, which crawls the actual dependency graph and figures out, for each package, exactly what’s blocking it.
Here’s what the scoreboard looked like before we started:
| Category | Count | % |
|---|---|---|
| Already on Bazel | 24 | 24% |
| Blocked by dep chain (torch) | 45 | 45% |
| Blocked by excluded externals | 22 | 22% |
| Blocked by C++ code | 4 | 4% |
| Manually excluded | 5 | 5% |
The 22 “blocked by excluded externals” are mostly packages that directly import torch — torch is in the excluded-externals list because Bazel can’t resolve it. The 45 “blocked by dep chain” are transitively blocked — they don’t use torch directly, but they depend on something that depends on something that eventually reaches it.
Together, that’s 67 packages — two thirds of the monorepo — sitting behind the torch wall.
The key insight: this file is machine-generated from the real dependency graph. No human guessing, no “I think package X needs torch.” The script crawls pyproject.toml files, resolves transitive dependencies, checks what’s available in Bazel, and categorizes everything. This is crucial. Manual tracking doesn’t scale past ~20 packages, and gut-feel estimates are always wrong in dependency graphs.
The Algorithm: Wave-by-Wave Cascade
The cascade analysis is a fixed-point computation. Simple in concept, surprisingly revealing in practice.
Setup
Start with an available pool — the set of packages that are currently on Bazel:
Available pool (start): 24 packages
mai_nano, lib/serialization, lib/maidas_client, ...
Wave 0: The Direct Unblock
Add the 20 packages that are blocked only by torch (the “excluded externals” group minus a few blocked by other things). If our fix makes torch available, these become immediately unblockable:
Wave 0: +20 packages (direct torch deps)
mai_config, mai_layers, mai_numerics, mai_preprocessors,
mai_trainer, mai_distributed, mai_multimodal, ...
Available pool: 44 packages
Wave 1+: The Cascade
Now scan every remaining blocked package. For each one, check: are ALL of its blockers now in the available pool? If yes — it’s unblocked. Add it to the pool and scan again.
Wave 1: Scan all blocked packages
✓ conversation — needed mai_config + mai_preprocessors → both available now
✓ mai_cluster — needed mai_config → available now
✗ lib/oai_api — needs lib/bus → still blocked (C++)
✗ common_utils — needs lib/oai_api → still blocked
Wave 1: +2 packages (conversation, mai_cluster)
Available pool: 46 packages
Wave 2: Scan again
✓ ci_harness — needed conversation → available now
✓ data_apis — needed conversation + mai_cluster → both available
✓ mai_api_utils — needed mai_cluster → available
✓ mai_prompts — needed conversation → available
✓ prompt_registry — needed conversation → available
✓ launcherv2 — needed mai_config + mai_cluster → available
✓ precook — needed mai_config + conversation → available
Wave 2: +7 packages
Available pool: 53 packages
Wave 3: Scan again
✓ mai_safety_api — needed mai_prompts + mai_api_utils → both available now
Wave 3: +1 package
Available pool: 54 packages
Wave 4: Scan again
(no new packages unblocked)
→ Fixed point reached. Done.
The Full Cascade
Wave 0: +20 (direct torch deps) ████████████████████
Wave 1: +2 (conversation, mai_cluster) ██
Wave 2: +7 (downstream cascade) ███████
Wave 3: +1 (mai_safety_api) █
─────────────────────
Total: +30 packages unblocked 30 packages → 24+30 = 54/101
From 24% to 53% Bazel coverage. More than doubling.
Poison Pills: The Ceiling You Can’t Fix
35 packages remain blocked even after torch is resolved. Why?
Because of poison pills — packages that are permanently blocked and propagate that blockage to everything downstream.
The biggest one:
lib/bus (C++/Cython) ← permanently blocked
└─ lib/oai_api ← poisoned
└─ common_utils ← poisoned
├─ research_utils ← poisoned
├─ launcher ← poisoned
└─ ... ~28 more ← all poisoned
lib/bus contains C++ code built with Cython. It can’t be built by rules_python. That’s a hard wall, not a configuration problem. And because lib/oai_api depends on it, and common_utils depends on lib/oai_api, and everything old depends on common_utils — you get a ~28-package contamination chain.
Other poison pills:
mai_kernels— CUDA C++ code (scikit-build-core + CMake)simplertransformer— same storylib/chz— manually excluded (complex metaprogramming that breaks Bazel’s analysis)fasttext— excluded external dep (C++ library)
The key insight here: poison pills set your ceiling. No matter how cleverly you solve the torch problem, these 35 packages can’t move until someone either rewrites lib/bus in pure Python, removes the common_utils dependency chain, or teaches Bazel to build C++ extensions. Each of those is a separate, significant project.
This is useful information! It tells you: don’t over-invest in the torch fix hoping to get to 80% coverage. Your ceiling is ~65% until the C++ problem is addressed. Prioritize accordingly.
The Path A Story: Why Analysis Comes Before Engineering
Before we settled on Option C (system Python toolchain in Docker), we tried something simpler. We called it Path A.
Path A: Make mai_config importable without torch. Since mai_config was the gateway blocker — the single config package that dragged torch into 50+ transitive dependents — what if we just made torch a lazy import? You could import mai_config and only hit the torch dependency if you called certain GPU-specific functions.
Sounds elegant, right? One surgical change, a few if TYPE_CHECKING guards, and the gateway opens.
We ran the cascade analysis on Path A. The result:
1 package unblocked.
Just mai_cluster. That’s it.
Why? Because most packages that depend on mai_config also have their own direct torch dependency. Making mai_config torch-free doesn’t help mai_layers — it imports torch directly for tensor operations. It doesn’t help mai_trainer — it uses torch.nn all over. It doesn’t help mai_distributed — it’s literally a distributed training library.
The gateway blocker narrative was wrong. Or rather, it was correct structurally (mai_config is a bottleneck in the graph) but misleading practically (removing it as a bottleneck doesn’t unblock the things behind it, because they’re blocked for their own reasons too).
Path A analysis:
mai_config becomes torch-free
→ mai_cluster: unblocked (only needed mai_config's torch)
→ mai_layers: still blocked (own torch dep)
→ mai_trainer: still blocked (own torch dep)
→ conversation: still blocked (needs mai_preprocessors, which needs torch)
→ ... everything else: still blocked
Total unblocked: 1
One package. After all that work refactoring mai_config’s imports.
This is why you run the analysis before writing the code. Path A would have taken days of careful refactoring, testing across 50+ downstream packages, and convincing the mai_config owners to accept lazy imports in their hot path. All for 1 package.
Option C — making torch available to all Bazel packages via the system Python toolchain — unblocks 30. The cascade analysis made the choice obvious.
The Methodology, Generalized
This isn’t specific to torch or Bazel. The wave cascade works for any “if we fix X, what unblocks?” analysis in a dependency graph. Here’s the general algorithm:
def cascade_analysis(
all_packages: set[str],
available: set[str],
blocked: dict[str, set[str]], # package → set of blockers
newly_unblocked: set[str],
) -> list[set[str]]:
"""Simulate wave-by-wave unblocking.
Returns a list of waves, where each wave is the set of
packages that become unblockable in that round.
"""
pool = available | newly_unblocked
waves = [newly_unblocked]
while True:
wave = set()
for pkg, blockers in blocked.items():
if pkg not in pool and blockers <= pool:
wave.add(pkg)
if not wave:
break
pool |= wave
waves.append(wave)
for pkg in wave:
del blocked[pkg]
return waves
The critical input is accurate blocker data. In our case, that comes from find_bazel_ready_packages.py crawling real pyproject.toml files. If you’re doing this for a different migration (say, Python 2→3, or moving off a deprecated framework), you need the equivalent — a script that can tell you, for each unit, exactly what’s blocking it.
Key Takeaways
1. Static analysis isn’t enough — you need the full graph. Counting “packages that import torch” gives you 20. The cascade gives you 30. That’s a 50% undercount. In a monorepo, transitive dependencies dominate.
2. Poison pills set your ceiling.
No matter how well you solve torch, lib/bus (C++) caps you at ~65%. Know your ceiling before you set your target.
3. Wave analysis shows diminishing returns. Wave 0 gives you 20 packages. Waves 1-3 give you 10 more. Each wave is smaller. This helps you decide: is the additional engineering worth the marginal unblocking?
4. Run the analysis before writing code. Path A looked great on a whiteboard. The cascade showed it unblocks 1 package. Option C unblocks 30. That’s a 30x difference in impact for comparable engineering effort. We would have wasted weeks without this analysis.
5. Machine-generate your migration status. The cascade analysis is only as good as its input data. A hand-maintained spreadsheet of “which packages need what” drifts within days. A script that crawls the real dependency graph is always current.
Next up: the actual implementation of Option C — registering Docker’s system Python as a Bazel toolchain, the py_venv_test trick, and how we split CI into torch and non-torch jobs. That’s the engineering. This was the math that told us the engineering was worth doing.