Bazel Migration: Benchmarking Bazel vs the Status Quo

I wanted a number. One clean number that says “Bazel’s test selection is X times more precise than find_changed.” Something I could put on a slide, show at a team meeting, and close the case.

So I built a head-to-head comparison tool. Run both test selectors on the same scenarios, produce a table, quantify the gap. I expected the data to say “Bazel runs fewer tests, clear win.” Instead, the tool told me my premise was wrong in a fundamental way. And the real story is more interesting than the one I was trying to tell.

The hypothesis

Quick context: I’m working on Bazel migration for a ~100-package Python monorepo. We have find_changed – a ~2,650-line Python system that decides which tests run on every PR. The pitch for migrating test selection to Bazel’s rdeps has always been: “Bazel is more precise because it reasons about targets, not packages.”

I’d been saying this in blog posts. I’d said it in the roadmap post. “Change one file in mai_layers, and every test in mai_layers runs. Every test in every package that depends on mai_layers runs. That’s the smallest unit find_changed can reason about – the package.”

That framing made the comparison tool easy to design. Count test files per package for find_changed, count py_test targets via bazel query rdeps(...) for Bazel, compare. Simple.

The first table (wrong)

My first version simulated find_changed by counting all test files in a package’s test directory. Because that’s what I thought it did – package-level selection.

The data looked great for the Bazel pitch:

“Look! For core libraries, Bazel selects 6-11x MORE tests – it catches cross-package dependencies that find_changed misses entirely! And for the ci package, Bazel is even more precise – 36 targets vs 44 test files.”

I was ready to write it up. Put it in a deck. Share it with the team.

Except those 11 tests for mai_nano? That was only counting tests inside mai_nano’s own directory. I was comparing find_changed’s within-package tests against Bazel’s cross-package transitive closure. Apples to entire fruit baskets.

The correction

When I actually read find_changed’s source code – properly, not just skimming – I discovered my fundamental premise was wrong.

find_changed is NOT package-level.

The tests and tests-by-package commands use FileTreeDiffChecker, which:

1. Builds a file-level import graph via ruff analyze graph
2. AST-scans every Python file for external distribution imports
3. Does memoized DFS through the file-level import graph – including cross-package imports

So when mai_nano/repo.py changes, find_changed doesn’t just select tests in mai_nano. It follows the import graph across package boundaries. caas_client imports mai_nano? Those tests get selected. observability imports something that imports mai_nano? Selected. It walks the full cross-package import graph. Just like Bazel does.

The tests_by_package command name is misleading – it groups results by package for the CI matrix, but the underlying selection is file-level.

My “11 tests for find_changed vs 71 for Bazel” was wrong because I only counted tests in mai_nano’s own directory. The correct number is that both tools select roughly the same set of ~71 tests across the whole repo.

Three blog posts I’d already written had this wrong framing baked in. The entire “package-level vs target-level” narrative I’d been building? Based on a misunderstanding.

This is the part where I’m supposed to feel embarrassed. But honestly – building the comparison tool is exactly how I discovered the error. If I’d never tried to quantify the gap, I’d still be making the wrong argument.

The real precision gap

So if both tools walk transitive import graphs… where’s the actual difference?

It’s smaller and different than I thought. Three places:

1. Within-package precision

When you change one file in a big package, Bazel knows which specific test targets depend on that file via explicit BUILD deps. find_changed uses ruff’s inferred import graph, which is close but not identical.

The CI tooling case shows this:

# Bazel: how many test targets in the ci package?
$ bazel query 'kind("py_test", //ci/...)' | wc -l
36

# find_changed: how many test files in ci's test directory?
$ find ci/tests -name "test_*.py" | wc -l
44

Those 44 test files are what find_changed considers when anything in the ci package changes. Bazel would only run the subset that actually depends on the changed file – potentially 36 or fewer. The within-package over-selection is real, but it’s a 1.2x difference, not a 6x difference.

2. Non-Python dependencies

This is where find_changed is strictly better.

find_changed has escape hatches for every kind of non-Python dependency that matters in our repo. Bazel’s Python rules see Python. That’s it.

Bump numpy from 1.24 to 1.25 with zero code changes? find_changed retests everything that depends on numpy. Bazel sees @pypi//numpy and shrugs – same target, no change, cache hit. Except the behavior might be different. find_changed is right here.

3. Coverage

find_changed covers all ~100 packages. Every one. Bazel covers ~25 migrated packages – about 167 py_test targets out of 1,000+ total test files in the repo.

For the 75 packages Bazel doesn’t know about, find_changed is the only game in town.

The blind spot table

Here’s what each tool catches that the other misses:

Neither tool dominates. They’re strong in different dimensions.

What the comparison tool actually showed me

The corrected comparison is boring in the best way:

The dramatic 6-11x ratios were artifacts of my broken simulation. The real ratios are close to 1:1 for cross-package selection, with a small within-package precision gap.

The one scenario where Bazel genuinely selects fewer tests is within large packages – where BUILD file granularity lets it skip tests that find_changed’s file-level graph includes. That’s real, but it’s a ~1.2x factor, not the 6x I was advertising.

The actual pitch

So if the precision gap is small, why migrate at all?

Because the pitch was never really “Bazel runs fewer tests.” That was my wrong hypothesis talking. The real pitch is convergence.

Both tools already walk transitive import graphs. The difference is in the graph source:

• find_changed: ruff analyze graph + AST scanning + manual config files
• Bazel: explicit deps in BUILD files, generated by Gazelle

Two independent graphs of the same codebase. They disagree on details. They always will, as long as both exist. (We got the full Bazel dep graph covering all packages, but the two graphs still diverge on non-Python edges.)

The path forward:

1. Migrate find_changed’s non-Python dep knowledge INTO Bazel (filegroups, data attributes, version tracking)
2. As Bazel migration completes, the two graphs converge into one
3. End state: retire ~2,650 lines of custom Python, replace with ~20 lines of bazel query

Not because Bazel is “more precise.” Because maintaining one graph is better than maintaining two.

And Bazel’s graph enables things find_changed structurally can’t do:

• Scoped pyright – rdeps to find affected packages, type-check only those
• GPU test targeting – attr(tags, "torch", rdeps(...)) replaces a manually-maintained YAML list
• Cached test results – same inputs = skip entirely, not just “run fewer tests”
• Hub/leaf analysis – fan-in counts, critical paths, decoupling opportunities

These are covered in What a Full Dep Graph Actually Unlocks – though that post needs a correction on the find_changed characterization. (This post is that correction.) And the graph stays accurate because BUILD file drift detection enforces it in CI.

The meta-lesson

I built a tool to prove my hypothesis. The tool disproved it. That’s the best possible outcome of building a comparison tool – finding out you were wrong before you presented wrong data to your team.

The narrative I was constructing – “find_changed is coarse, Bazel is precise, look at these dramatic ratios” – was clean and compelling. It was also wrong. The real narrative is messier: both tools are smarter than I gave them credit for, the precision gap is smaller than expected, and the case for migration rests on architectural convergence, not raw numbers.

Messier narratives are harder to sell. They’re also more honest. And honesty compounds – the team trusts the next number you show them because you corrected the last one.

This is part of a series on CI and build systems in a Python monorepo. The find_changed deep dive: Inside find_changed. The two-graph framing: Your Monorepo Has Two Dependency Graphs. The original roadmap (with the framing this post corrects): What a Full Dep Graph Actually Unlocks. The meeting that started it all: When Homegrown Tools Hit the Ceiling.