We migrated 15 packages to Bazel across two PRs. The BUILD files took a day. The debugging took a week. This is the story of everything that went wrong.
The strategy
Our monorepo has ~200 Python packages. Migrating them all to Bazel at once is suicide. So we broke it into phases, driven by the dependency graph.
Phase 0 was the proof-of-concept: 5 packages with minimal dependencies, chosen specifically because they should be easy. The goal wasn’t the packages themselves. It was proving that the pipeline works: Gazelle generates BUILD files, CI runs Bazel jobs, tests pass, and we can merge without breaking anything for everyone else.
Phase 1 was the first batch that actually matters: 10 packages including lib/oai_api (which unblocks 4 downstream packages) and posttraining/prompt_registry (which unblocks common_utils). The dependency graph tells you what to migrate next. You pick the nodes that unblock the most edges.
Simple on a whiteboard. Messy in practice.
Phase 0: the “easy” batch
PR #28693 started clean. Five packages, all flagged as “ready” in our migration tracker. Gazelle generated the BUILD files. Local builds passed. Push, open PR, wait for CI.
First failure: missing fortknox dependency. One of the packages imported it transitively through a chain that Gazelle couldn’t trace. Gazelle sees your import statements, but it doesn’t see the runtime plugin loading that fortknox uses. Manual fix. One commit.
Second failure: untracked test data files. Tests passed locally because the files existed in the working tree. Bazel’s sandbox doesn’t have your untracked files. You either add them to data in the BUILD rule, or you find out in CI. We found out in CI.
Third failure, and this one was educational: tests that import torch. Not directly. Through mai_config, which imports mai_torch_utils, which imports torch. The test file says import mai_config and that’s it. Looks clean. But under py_test, there’s no torch in the sandbox. The error message doesn’t say “torch not found.” It says something like ModuleNotFoundError: No module named 'mai_accelerate_utils', three layers deep in the import chain. You stare at it, check the BUILD file, check the deps, everything looks right. Then you realize: this test needs py_torch_test, our custom macro that includes the torch dependency closure.
Three fix commits for the “easy” batch. Merged on April 16. Migration counter: 36 to 41 configured packages.
Phase 1: where the archaeology starts
PR #29379 was 48 files changed across the monorepo. Ten new packages. And a commit history that reads like a debugging journal.
Gazelle drift
Here’s something you don’t expect: packages that are already listed in GAZELLE_DIRS but have no BUILD files committed. mai_cluster and mai_logger were in the Gazelle config. Gazelle was supposed to be managing them. But their BUILD files were either never committed or went stale months ago when someone refactored without running Gazelle.
You discover this when you add a new package that depends on them. Bazel resolves the dependency, looks for the BUILD file, doesn’t find it (or finds a stale one), and fails. The error points at your new package. The problem is in a package someone else “migrated” six months ago.
Similarly, codeshield and soft_distillation were in GAZELLE_DIRS but literally had no BUILD files on disk. Ghost entries. We had to generate and commit their BUILD files as part of our PR, even though they weren’t in our migration batch.
This is the pattern: every migration PR is also an audit of everything the migrated packages touch. You’re not just building forward. You’re backfilling gaps that were invisible until now.
The .bazelignore surgery
Our .bazelignore had a single line: posttraining. Ignore the whole directory tree. Quick and dirty, from the early days when nothing in posttraining was migrated.
Phase 1 included posttraining/prompt_registry. So we couldn’t ignore all of posttraining anymore. But we also couldn’t remove the ignore line entirely, because most posttraining subdirectories weren’t migrated yet and had Python files that would confuse Gazelle.
The fix: replace the blanket posttraining ignore with individual lines for each unmigrated subdirectory. One per line. Miss one, and Gazelle tries to parse an unmigrated package, generates a broken BUILD file, and every Bazel target in the tree fails.
This required listing every subdirectory, checking which ones were migrated, and writing the ignore list by hand. Not hard. Just tedious, error-prone, and the kind of thing where you only find your mistake after a 12-minute CI round trip.
Broken tests vs. broken builds
This is the ownership boundary that took us a while to internalize.
When CI fails on a migrated package, there are two possibilities:
- Bazel can’t build or run the test. Missing dep, wrong macro, sandbox issue. That’s our bug.
- The test ran fine, but its assertions failed. The test itself is broken. That’s the package owner’s bug.
During migration, you can’t block on #2. If a test was already failing before migration (and it often was, just nobody noticed because it wasn’t in any CI job), that’s not a Bazel problem. Tag it manual so Bazel skips it, file a ticket for the owner, move on.
The temptation is to fix the test. Don’t. You’ll be fixing other people’s tests forever, and the migration will stall. The BUILD file is your scope. The test content is theirs.
The CI iteration loop
Here’s what Phase 1’s commit history actually looks like:
- Initial migration: 10 packages, BUILD files generated
- Fix: Gazelle drift in
mai_clusterBUILD - Fix: Gazelle drift in
mai_loggerBUILD - Fix: add missing BUILD for
codeshield - Fix: add missing BUILD for
soft_distillation - Fix: switch 3 tests from
py_testtopy_torch_test - Fix: tag 2 broken tests as
manual - Fix:
.bazelignoreindividual entries for unmigrated posttraining subdirs
Each fix takes ~15 minutes of CI. You push, wait, read the logs, figure out what the actual problem is (not what the error message says), fix it, push again. The PR commit history is the debugging log.
The label trap
This one is just mean. Our CI has a wait-for-label job for GPU tests. If the run-gpu-tests label isn’t on the PR, the GPU jobs don’t run. They don’t fail. They get cancelled. The status shows up as cancelled, not skipped or missing label. The required gpu_tests_passed gate sees cancelled jobs and blocks the merge.
If you don’t know about this label, you will spend 30 minutes staring at “cancelled” GPU jobs wondering what went wrong. Then you add the label, but CI already ran, so you have to re-push to trigger a fresh run.
Lesson learned. Add run-gpu-tests immediately after opening the PR. Every time. No exceptions.
The meta-lesson
Migration work is 20% writing BUILD files and 80% archaeology.
Gazelle does the mechanical part. It reads your imports, it generates the rules. That part works. What Gazelle can’t do is tell you about the fortknox plugin loader, or the six-month-old BUILD file that was never committed, or the test that imports torch through three layers of indirection, or the .bazelignore entry that’s been hiding a broken directory tree.
Every one of those discoveries is an implicit assumption that was never written down. “This package needs torch at runtime.” “This BUILD file needs to exist.” “This test has been broken since January.” The migration doesn’t create these problems. It just makes them visible.
Phase 0 took us from 36 to 41 configured packages. Phase 1 will take us from 41 to 56. The counter ticks up slowly, but each tick means one more package where the dependency graph is explicit, the build is hermetic, and the implicit assumptions are now written in a BUILD file where you can actually see them.
That’s the real value. Not faster builds (though those are nice). The migration forces you to write down everything that was previously just vibes.