Bazel Migration: Reading Status Like a Map

There’s a script in our repo called find_bazel_ready_packages.py. Ryan wrote it. It generates a file called MIGRATION_STATUS.md — a machine-generated scoreboard that tells you, for every package in the monorepo, exactly where it stands and exactly what’s blocking it.

I’ve been using this script for weeks. I’ve re-run it after every change, watched the numbers shift, built wave-by-wave cascade analyses on top of its output. But I never actually sat down and read the code. Not “glanced at it.” Read it — traced the data flow, understood the reuse patterns, noticed what it’s good at and where it goes quiet.

So that’s what this post is. A walkthrough of one script that, more than any planning doc or JIRA board, tells you where a Bazel migration actually stands. What it reveals, what it misses, and what its output tells you about the road ahead.

The script, phase by phase

The whole thing is ~370 lines. Four phases, clean separation, and a reuse pattern that’s more interesting than the script itself.

Phase 1: Build the dependency graph

The first thing find_bazel_ready_packages.py does is not build its own dependency graph. It reuses one.

from ci.workspace_graph import (
    WorkspaceDistribution,
    build_distributions_from_worktree,
)

_, workspace_distributions = build_distributions_from_worktree(
    repo_root=repo_root,
    package_extra_deps_config=package_extra_deps_config,
    env=env,
)

build_distributions_from_worktree() lives in ci/workspace_graph.py — the same module that powers CI test selection. It reads uv.lock (the lockfile), parses it via ci.config_parsers.uv_lock.parse_uv_lock() into a structured UvLock pydantic model, resolves workspace members, evaluates conditional dependencies against a synthetic environment, and produces a typed graph of WorkspaceDistribution objects.

The synthetic environment is worth pausing on:

env = make_env("3.12", "x86_64")

That call creates a fake Linux/x86_64/Python 3.12 environment — not because we’re running on that, but because packaging.markers.Marker.evaluate() needs an environment to decide whether conditional dependencies apply. A dependency like torch ; sys_platform == 'linux' only resolves if the environment says “yes, I’m Linux.” The migration tool needs the Linux picture because that’s where CI runs.

The UvLock parser itself is wonderfully minimal — 39 lines of Pydantic:

class UvLock(BaseModel):
    manifest: Manifest = Field(default_factory=Manifest)
    packages: list[PackageEntry] = Field(alias="package")

def parse_uv_lock(lock_toml: str) -> UvLock:
    return UvLock.model_validate(tomllib.loads(lock_toml))

That’s the entire lockfile parser. tomllib handles TOML, Pydantic handles validation. No custom parsing.

Key insight here: The migration script’s first move is to piggyback on infrastructure that already existed for a different purpose. CI test selection needed a dependency graph. Migration status needs the same graph. Same data, different question.

Phase 2: Build the dist-to-path map

CI’s WorkspaceDistribution tracks packages by their canonical distribution name (mai-config → mai_config). But the migration tool needs to know where packages live on disk — it’s going to scan for BUILD files, check for C++ code, read pyproject.tomls.

def build_dist_to_path_map(repo_root: Path) -> dict[str, Path]:
    pyproject_toml = (repo_root / "pyproject.toml").read_text(encoding="utf-8")
    pyproject = load_pyproject(pyproject_toml)

    dist_to_path: dict[str, Path] = {}
    for member in pyproject.tool.uv.workspace.members:
        member_path = Path(str(member))
        # ... read each member's pyproject.toml, extract canonical name
        canonical = DistributionName(member_data.project.name).canonical
        dist_to_path[canonical] = member_path

    return dist_to_path

It reads the root pyproject.toml’s [tool.uv.workspace.members] list — the same list uv uses to discover workspace packages — and maps each canonical distribution name to its filesystem path.

DistributionName is a frozen dataclass that normalizes names per PEP 503: My-Package → my_package → my-package. This matters more than you’d think. Python packaging has historically been cavalier about naming — hyphens, underscores, mixed case, all “the same” package. Canonical normalization means the graph doesn’t accidentally split mai-config and mai_config into two nodes.

Phase 3: Classify every package

This is the core. For every package in the workspace that doesn’t already have a BUILD file, the script runs a decision tree:

For each package (sorted):
  ├── Already has BUILD.bazel?     → Skip (already migrated)
  ├── In MANUALLY_EXCLUDED?        → manually_excluded
  ├── Contains C/C++/CUDA/Rust?    → blocked_by_language
  ├── Depends on excluded external? → blocked_by_external
  ├── Any workspace dep unmigrated? → blocked_packages (dep-blocked)
  └── All deps migrated            → ready_packages

Order matters. The checks run top-to-bottom, and a package gets classified into the first bucket that matches. A package with both C++ code and a torch dependency lands in blocked_by_language, not blocked_by_external. This is a design choice — it surfaces the harder problem first.

Each check is its own function:

has_bazel_config() recursively scans the package directory for any BUILD.bazel or BUILD file. Not just at the root — list(package_path.rglob("BUILD.bazel")). Some packages have nested BUILD files (like lib/blobfile/src/blobfile/BUILD.bazel), and any of those count.

MANUALLY_EXCLUDED is a hardcoded dict mapping workspace paths to exclusion reasons:

MANUALLY_EXCLUDED: dict[str, str] = {
    "lib/excel_evaluation": "complex native deps",
    "caas_server": "",
    "dataprocessing/audio_ops": "gazelle detects optional torch import, ...",
    "dataprocessing/resumable_ray_pipeline": "gazelle detects optional torch import, ...",
}

(We’ll come back to that empty string for caas_server.)

has_non_python_code() scans src/, csrc/, kernels/, 3rdparty/ for file extensions: .c, .cc, .cpp, .cu, .rs, .go, plus CMakeLists.txt. If any are found, the package is classified as language-blocked. This is the “you can’t get here from Python-only Bazel rules” detector.

get_excluded_external_deps() reads the package’s pyproject.toml and checks all dependency sections — [project.dependencies], [project.optional-dependencies], [dependency-groups] — for any package in EXCLUDED_EXTERNAL_PACKAGES. Which, for the entire history of this script, has been exactly one entry:

EXCLUDED_EXTERNAL_PACKAGES: set[str] = {
    "torch",
}

The dep-blocked check iterates over workspace dependencies (other monorepo packages) and checks whether each has a BUILD file. If any don’t, the package is blocked by those specific dependencies.

for dep in workspace_dist.dependencies:
    if isinstance(dep, WorkspaceDistribution):
        dep_canonical = dep.name.canonical
        if dep_canonical not in bazel_configured:
            blocking_deps.append(dep_label)

The isinstance check is filtering for workspace deps specifically — the dependency list also contains ExternalDistribution objects (PyPI packages), but those aren’t relevant for migration readiness. You only care about monorepo deps that haven’t been migrated yet.

Phase 4: Print the markdown report

The output phase is straightforward: categorized lists with blocking reasons, then a summary. But there’s one line at the bottom that I appreciate:

not_analyzed = (
    len(workspace_distributions)
    - len(bazel_configured)
    - len(ready_packages)
    - len(blocked_packages)
    - len(blocked_by_language)
    - len(blocked_by_external)
    - len(manually_excluded)
    - len(skipped_no_path)
)
if not_analyzed != 0:
    print(f"- **Not analyzed (unexpected):** {not_analyzed}")

Every package must land in exactly one bucket. If the buckets don’t sum to the total, something is wrong. This is a conservation check — the script accounts for every package or tells you it didn’t. Machine-generated status that also validates itself. Compare this to a spreadsheet where row count doesn’t match reality and nobody notices for three sprints.

The companion tooling

The migration status script doesn’t work alone. It’s part of a small tooling ecosystem — four scripts that attack the migration from different angles, all sharing the same underlying data model.

check_bazel_coverage.py — the post-migration auditor. After you’ve migrated a package and generated BUILD files, this script finds .py files that exist on disk but aren’t tracked by any Bazel target. It parses GAZELLE_DIRS from bazel/BUILD.bazel, scans for Python files, queries bazel query labels(srcs, ...), and diffs. It also detects configuration contradictions — a package in GAZELLE_DIRS but also in .bazelignore, which is “migrate me” and “ignore me” at the same time.

generate_manual_builds.py — the graph filler. This is the clever one. It generates stub BUILD files tagged manual for every unmigrated package. The stubs don’t build, don’t test, don’t do anything — except make the package visible to bazel query. (I wrote a whole post about this.) It classifies every dependency into workspace, system (torch/triton/vllm → //bazel/stubs:{name}), or external (@pypi//{name}), then emits a py_library with the right labels. The full monorepo dep graph becomes queryable even when half the packages aren’t actually migrated.

show_affected.sh — the blast radius calculator. Given a Bazel package path, it runs bazel query rdeps(//..., $PACKAGE) and tells you how many test targets are affected. Simple, but the output is immediately useful: “This package change affects 60/986 targets — 94% skip rate.”

The layered reuse pattern across these tools is the thing worth noticing:

1. uv.lock is the source of truth for all dependencies
2. ci/workspace_graph.py parses it into a typed graph (shared by CI test selection AND migration tooling)
3. find_bazel_ready_packages.py adds Bazel-specific classification on top
4. generate_manual_builds.py generates a Bazel representation of that same graph
5. check_bazel_coverage.py validates the result post-migration

Five tools, one data model, five different questions. That’s the right kind of tooling ecosystem — shared infrastructure with specialized consumers. Nobody reimplemented a lockfile parser.

The critical commentary

Now the opinionated part. I’ve been staring at this script’s output for weeks. Here’s what it’s good at, and where it falls short.

Torch as EXCLUDED_EXTERNAL_PACKAGES

One line of code:

EXCLUDED_EXTERNAL_PACKAGES: set[str] = {"torch"}

One entry in a set literal. This single entry blocked 21 packages directly and cascaded to block ~45 more transitively. Sixty-six percent of the monorepo, held hostage by one element in a hardcoded set.

The fix (PR #21169 — system torch via Docker’s local Python toolchain) was non-trivial. But the migration tooling made the impact immediately visible. Before I wrote a single line of fix code, I could run find_bazel_ready_packages.py, stare at the “Blocked by Excluded External Dependencies” list, and know: this is the lever. Pull it, and 30 packages move.

That kind of clarity — “one decision, quantified impact, before you commit” — is what separates computed status from vibes-based planning.

The cascade problem

The script classifies packages into buckets. Good. But it doesn’t compute leverage.

Forty-seven packages are “dep-blocked” — they depend on something that isn’t migrated yet. The script lists each one with its blocking deps. But what it doesn’t tell you is that just four packages — conversation, common_utils, lib/oai_api, mai_config — gate roughly half of all blocked packages.

If you look at the output naively, you see 47 blocked packages and think “big problem, evenly distributed.” The reality is that the blockage is extremely concentrated. A handful of choke points cascade through the graph.

I ended up building wave analysis on top of the script’s output to surface this. It wasn’t hard — fixed-point computation, iterate until stable. But the fact that I had to build it separately means the script is answering “what’s blocked?” without answering “what should I unblock first?”

The difference between a status report and a strategy tool is exactly that: leverage analysis. Where do you spend one unit of effort to get ten units of unblocking?

lib/bus is the real wall

The script correctly identifies lib/bus as blocked by non-Python code (C++/Cython). What it can’t tell you is that lib/bus transitively blocks ~28 packages through the lib/bus → lib/oai_api → common_utils → everything chain.

This is a fundamentally different kind of blocker than torch. Torch was a dependency resolution problem — the package exists, it works, we just needed to make Bazel aware of it. C++/Cython is a build system problem. You need rules_cc, CUDA toolchains, Cython compilation rules — infrastructure that doesn’t exist yet and is genuinely hard to build.

The migration tooling makes this transition visible. Run the script before and after the torch fix: the “blocked by external” list shrinks dramatically, but the “blocked by non-Python code” and “blocked by dependencies” lists barely change. You can literally watch the character of the migration shift from “Python dependency management” to “native code integration.”

Manual exclusions feel like tech debt

MANUALLY_EXCLUDED: dict[str, str] = {
    "lib/excel_evaluation": "complex native deps",
    "caas_server": "",
    "dataprocessing/audio_ops": "gazelle detects optional torch import, ...",
    "dataprocessing/resumable_ray_pipeline": "gazelle detects optional torch import, ...",
}

Four packages. Two have real reasons. One has a reason that might be fixable now that torch is available in Bazel. And caas_server has… an empty string.

Manual exclusions in a dict literal are fine for four packages. They’re a problem at twenty. Every exclusion without a reason is a future “why is this here?” question that nobody can answer. Every exclusion with a reason should have a link to an issue — a trail back to the decision and forward to the eventual fix.

No wave analysis built in

The script classifies packages into buckets but doesn’t compute the cascade effect: “if I unblock X, what becomes ready next?” That analysis — simulating the removal of a blocker and computing which dep-blocked packages become unblocked — is exactly the question a migration planner asks every week.

This isn’t a criticism of Ryan’s design. The script does exactly what it says: find packages ready right now. But the gap between “what’s ready now” and “what becomes ready if I fix this one thing” is where all the strategic decisions live.

The graph after the torch fix

After PR #21169 landed, I ran the cascade analysis I’d built on top of the migration status output. Here’s what the torch fix unlocked, wave by wave:

Wave 0 — Direct unblock (20 packages). Everything that was blocked only by torch: mai_config, mai_layers, mai_trainer, mai_distributed, mai_multimodal, rocket_lib, sgl_server, st_server, tool_calling, verifiable_data, and more. These all had BUILD files waiting or were immediately ready for Gazelle.

Wave 1 — First cascade (2 packages). conversation needed mai_config + mai_preprocessors, both now available. mai_cluster needed mai_config. Two packages, but conversation is one of the highest-fan-out packages in the monorepo.

Wave 2 — Second cascade (7 packages). ci_harness, data_apis, mai_api_utils, mai_prompts, prompt_registry, launcherv2, precook. All gated by Wave 0 or Wave 1 packages.

Wave 3 — Third cascade (1 package). mai_safety_api, unblocked by Wave 2.

Total: 30 packages. From 27/104 on Bazel to potentially 57/104 — that’s 26% to 55%.

But here’s the part the numbers don’t show: the character of the remaining 35 packages.

The torch fix was the phase transition. Before it, the migration was a Python dependency management problem. After it, the remaining blockers are almost entirely native code — C++, CUDA, Cython. Different tooling, different expertise, different timeline.

Run find_bazel_ready_packages.py before the torch fix and after. The output changes shape. The “blocked by external” section empties out. The “blocked by non-Python code” section stays exactly the same. The “blocked by dependencies” section shrinks but the remaining entries all trace back to lib/bus or lib/chz.

The script doesn’t tell you about phase transitions in words. It tells you by what doesn’t change when you make a big move.

What this taught me about migration strategy

Reading find_bazel_ready_packages.py carefully — not just running it, but understanding how it builds its picture — taught me something about how to think about large migrations.

Computed status is more honest than maintained status. A human-maintained migration tracker reflects what someone believes the status is, filtered through optimism and staleness. A script that crawls pyproject.toml files and checks for BUILD files reflects what is. It doesn’t care about your timeline. It doesn’t round up.

The classification cascade reveals the real constraints. The script checks for non-Python code before checking for excluded externals before checking dep-blocked. That ordering isn’t arbitrary — it surfaces the hardest problems first. If a package has C++ code AND a torch dep, the C++ problem is the one that matters, because fixing torch doesn’t help you.

The tools you build to measure progress become the tools that plan the next move. The migration status script was built to generate a scoreboard. It turned out to be the foundation for cascade analysis, wave planning, and impact quantification. The best migration tooling isn’t one script that does everything — it’s a shared data model that multiple scripts can query for different purposes.

Watch what doesn’t move. The exciting number is the one that goes up — 27 packages to 57. The important number is the one that stays the same — 35 packages still blocked, all by native code. If you only look at the progress line, you miss the wall approaching.

The script is 370 lines. It took Ryan maybe a day to write. It’s been the single most useful artifact in a migration that’s touched 104 packages. Not because it does anything clever — the algorithm is a sorted loop with five if statements. But because it asks the right question at the right layer of abstraction, and it gives you an answer you can actually trust.

Hand-maintained status lies to you gently. Computed status tells you exactly where you stand — including the parts you don’t want to hear.