Graduating Test Selection from find_changed to Bazel

Last post I told the story of a meeting where we decided to stop investing in our homegrown test selection tool. This post is the sequel nobody asked for: okay, but how does the handoff actually work?

Because you can’t just delete find_changed and type bazel test //.... Not when Bazel only knows about 8% of your packages. That’s like firing your translator before the new one has learned the language.

The transition is gradual, and the rate of graduation is proportional to one thing: how much of the dependency graph Bazel can see.

What find_changed Actually Does

For the uninitiated: we have a monorepo with ~95 Python packages. When a PR lands, something needs to answer the question which tests should run? Running everything is slow. Running nothing is how you break main on a Friday evening and ruin someone’s weekend.

find_changed is the homegrown tool that answers this question. It:

1. Looks at what files changed in a PR
2. Maps changed files to affected packages
3. Traces which test packages depend on those packages
4. Outputs the list of tests CI should execute

It’s essentially a dependency analysis engine, and the analysis logic is actually decent. The tool knows how to trace imports, follow package boundaries, and figure out the blast radius of a change.

Where it breaks isn’t in the analysis. It’s in the specification.

People have to declare that their test depends on a config file, a data fixture, a module loaded via some dynamic import. Researchers writing model training code are not thinking about CI dependency graphs. They shouldn’t have to be. But someone has to, and with find_changed, the answer to “who catches missing deps?” is “the oncall engineer, three weeks later, at 11 PM.”

The other problems compound from there:

• Import-to-package mapping was missing — one of our engineers spent weeks building the logic that maps Python import names to monorepo package names. This is exactly what gazelle’s Python plugin does automatically, for free, maintained by a community. We rebuilt a subset of gazelle on purpose.
• No sandbox — if find_changed misses a dependency, the test still passes. It can see the entire filesystem. The missing dep is invisible until someone changes the undeclared dependency and the test doesn’t run. In Bazel’s sandbox, a missing dependency means the test fails immediately. Missing deps are silent in find_changed, loud in Bazel.
• Performance — it runs in about a minute now (after optimization work), which is fine. But Bazel’s bazel query with a materialized dep graph would be near-instant.

What Bazel Gives You for Free

Once a package is migrated to Bazel, you get:

• Automatic dependency detection via gazelle — it scans your imports, generates BUILD files, no manual declaration needed. The entire class of “someone forgot to declare a dep” shrinks dramatically.
• Sandbox enforcement — every test runs in an isolated environment. If your BUILD file doesn’t list a dependency, your test can’t see it. Period. You discover missing deps at authoring time, not during oncall.
• Incremental testing — bazel test //... only runs tests whose transitive dependencies changed. This is the core value proposition. It’s not a heuristic. It’s a mathematical guarantee based on the dep graph.
• Remote caching — if a test’s inputs haven’t changed, it doesn’t run at all. Cache hit. Done. This is the difference between “we run fewer tests” and “we run zero tests when nothing changed.”
• Global rules — “every source file must belong to a target” is enforceable in Bazel. In find_changed, you can have orphaned files that nothing tracks, and you’d never know until they cause a problem.

This isn’t theoretical. These properties fall out of Bazel’s architecture. The dep graph is the build system. Test selection is just a query against it.

The Graduation Table

Here’s the thing I keep coming back to. The delegation from find_changed to Bazel isn’t a binary switch. It’s a slider, and the slider position is determined by how many packages Bazel knows about:

Read that table from top to bottom and you’ll notice something: the jump from 33% to 85% is one PR.

That PR is what I’ve been calling Path A — making torch imports lazy in a core config package. Right now, mai_config imports torch at module level, which means every package that depends on mai_config (read: most of them) transitively depends on torch. And torch is system-installed via Docker, not in uv.lock, which means Bazel can’t resolve it through its normal pipeline.

Make the import lazy, and ~50 packages that don’t actually use torch stop depending on it. They become Bazel-eligible overnight.

That single PR would move us from “Bazel handles a third of our packages” to “Bazel handles almost everything, and find_changed is just mopping up the GPU-heavy corners.”

The Risk Nobody Wants to Talk About

Here’s where I get uncomfortable.

We decided to stop investing in find_changed. No new features, no improvements to the analysis code. The reasoning was sound: every hour spent improving find_changed is an hour not spent on Bazel migration, and Bazel gives you all of find_changed’s features for free once you’re there.

But “once you’re there” is doing a lot of heavy lifting in that sentence.

If Path A stalls — if the lazy import PR gets stuck in review, or reveals unexpected breakage, or just takes longer than expected — we end up in the worst of both worlds:

• find_changed stops getting better (because we decided to stop investing)
• Bazel only covers 33% of packages
• 67% of our test routing is handled by a tool nobody’s improving

That’s not a fun place to be. It’s the gap between “we decided to stop” and “the replacement is ready.” Every migration has this gap. The question is how long you spend in it.

This is why the team’s deep dive on accelerating Bazel migration matters. It’s not a planning exercise. It’s risk mitigation for a decision we already made.

What I’m Actually Doing About It

I’m attacking the torch blocker from three directions in parallel. Not because I’m thorough — because I genuinely don’t know which one will work, and finding out sequentially would take months I don’t have.

Approach 1: Lazy imports in mai_config (Path A)

The surgical option. Make torch imports in mai_config lazy — move them inside the functions that actually use them. This severs the transitive torch dependency for ~50 packages that import mai_config for config purposes but never touch a tensor.

This is the highest-leverage move. One PR, ~50 packages unblocked. But it requires auditing every torch import in mai_config, handling type annotations (via TYPE_CHECKING), and making sure nothing relies on torch being available at module import time. It’s code-change work, not infrastructure work. Ships faster, fails cheaper.

Approach 2: Add Linux CPU wheels to uv.lock (Option A from the architecture post)

Remove the sys_platform == 'darwin' gate on torch in pyproject.toml. Let uv lock resolve Linux CPU wheels. Now @pypi//torch has a valid target on Linux, select() works everywhere, and Bazel can build torch-dependent packages.

The tradeoff: the lockfile gets bigger (~800MB of CPU torch wheel metadata), every uv lock resolution has to consider Linux torch, and CI runners need to download a large wheel on first cache miss. But the conceptual model is the cleanest — torch is just a pip package everywhere.

Approach 3: http_archive a CPU torch wheel (Option B)

Download a specific CPU torch wheel as a Bazel external dependency. Wire it into @pypi//torch via the override mechanism. Precise, hermetic, but the override is global — and as I documented in the override comedy, global overrides and platform-specific behavior don’t play well together.

The bet: if any one of these approaches works, we unblock ~60 packages. That gets us to the 85% line on the graduation table. At that point, Bazel’s dep graph is complete enough to handle test-diff analysis for the vast majority of our codebase, and find_changed becomes a minor supplement for the GPU/C++ tail.

The timeline: honestly uncertain. Ryan and I are both part-time on this. But the part-time constraint is partly because we don’t have verification yet — once we confirm an approach works, it’s worth shifting focus to land it. The three-pronged strategy is as much about accelerating the verification as it is about hedging bets. Whichever one gets to “this works on a test package” first, that’s where we invest.

The Work That Transfers

One decision from the meeting was genuinely clever: keep adding missing dependencies when you find them.

This sounds like investing in find_changed, but it’s not. Dependency specification is the one piece of work that transfers directly to Bazel. Every missing dependency you declare in find_changed today is a dependency that needs to exist in a BUILD file tomorrow. The knowledge transfers. The format changes, but the understanding doesn’t.

It’s the difference between investing in the analysis engine (doesn’t transfer — Bazel replaces it entirely) and investing in the dependency knowledge (transfers — Bazel needs the same information, just expressed differently).

The Mental Model

I keep thinking about this transition as a trust transfer.

Right now, find_changed is the authoritative answer to “which tests should run?” It has the dep graph (imperfect, manually maintained, but real). It makes the routing decision. CI trusts it.

Bazel is building a parallel dep graph — more accurate, automatically maintained, sandbox-enforced. But it only covers 8% of the world.

The graduation happens package by package. Each migrated package is a piece of the dep graph that moves from find_changed’s jurisdiction to Bazel’s. At some point — probably around 85% — the balance tips. find_changed goes from “the system” to “the fallback for a few edge cases.” And eventually, it goes from “the fallback” to “that script we should probably delete but nobody wants to be the one to do it.”

The speed of this transition is entirely determined by the speed of the Bazel migration. Which is entirely determined by whether we can unblock the ~60 packages sitting behind the torch dependency wall. Three approaches running in parallel. Part-time, for now.

Once one of them clears verification, the timeline compresses fast. 85% coverage means find_changed goes from “the system” to “the thing we keep around for ten packages.” And at that point, we can start doing diff-based test selection in Bazel — the thing that makes all of this worth doing in the first place.

No pressure.

This is part of an ongoing series on migrating a GPU-heavy Python monorepo to Bazel. Previously: the meeting that changed the plan, how Python dependency resolution works in Bazel, and the migration battle plan.