There’s a moment in every homegrown tool’s life where it stops being the scrappy underdog solving a real problem and starts being the thing you spend Fridays apologizing for. Our dependency analysis tool hit that moment last week, in a meeting that started as an oncall retro and ended as a Bazel migration commitment.
This is the story of how a brutal oncall week made us stop polishing the tool we had and commit to the one we need.
The Tool
Our monorepo has ~95 Python packages. When a PR lands, something needs to answer the question: which tests should run? Running all of them takes forever. Running none of them is how you break main at 6 PM on a Friday.
The answer to this question was a homegrown tool called find_changed. It analyzes what files a PR touches, maps those files to affected test packages, and tells CI which subset of tests to execute. It’s the dependency analysis engine behind our entire test selection strategy.
And it worked. For a while.
The Oncall
Our oncall engineer had one of those weeks. The kind where you start Monday with a clean dashboard and end Friday questioning your career choices.
The headline problems:
- Three PRs merged that broke main. In each case,
find_changeddidn’t know about a dependency — the changed code affected tests that never ran during PR CI, because the dependency graph had holes in it. - Flaky tests were slow to disable. Test owners pushed back hard on disabling anything. “It’s a really important test that guards important code.” Which — sure. But a test that passes 70% of the time doesn’t guard anything. It just makes everyone ignore CI.
- Dependencies require manual declaration, and people don’t do it. If your test depends on a config file three directories over, someone has to explicitly tell
find_changedabout that relationship. Researchers writing model code are not thinking about CI dependency graphs. Why would they be? - Force merges happened. The oncall engineer himself force-merged a PR, breaking the rules he was supposed to enforce. (“I have no idea why I actually force merged it.”) Another teammate force-merged a change that looked safe from the diff but wasn’t. When the people running the system are bypassing the system, the system has a problem.
The Conversation
Our squad lead asked the question everyone had been circling: at what point do we shift our efforts from the homegrown dependency tracking to Bazel?
The oncall engineer — who knows find_changed better than anyone — gave the most nuanced answer I’ve heard in a meeting:
The analysis code isn’t where most misses happen. The tool’s logic for tracing imports and mapping changes to packages is actually decent. The failures come from dependency specification — a test doesn’t declare it depends on a config file, or a data fixture, or a module loaded dynamically. Someone would need to specify these dependencies whether they’re using find_changed or Bazel or anything else.
But Bazel already handles the import-to-package mapping. One of our engineers had spent weeks building the logic that maps Python import names to monorepo package names for find_changed. That’s exactly what gazelle’s Python plugin already does — automatically, maintained by a community, tested at scale. We rebuilt a subset of gazelle. On purpose.
And Bazel’s sandbox catches what find_changed silently misses. This is the killer difference. If you forget to declare a dependency in find_changed, nothing happens — the test still passes locally and in CI, because it can see the entire filesystem. You only discover the gap later, when a PR changes that undeclared dependency and the tests don’t run. With Bazel, if you forget a dependency, the test fails immediately in the sandbox. The missing dep is surfaced at authoring time, not three weeks later during someone’s oncall.
The squad lead put the meta-point cleanly: “We’re increasingly rebuilding large subsets of Bazel in our homegrown tools.”
The Decision
We made three commitments:
-
Stop investing in
find_changedlogic. No new features. No improvements to the analysis code. The oncall engineer admitted there was a whole list of improvements he wanted to make — better import tracing, smarter change detection — and he was telling himself no, Bazel will give us all of that for free. -
Keep adding missing dependencies when found. This is the one piece of work that transfers directly. Every missing dependency you declare today is a dependency that Bazel will also need. This work isn’t wasted.
-
Accelerate the Bazel migration. Our squad lead committed to a deep-dive at the next team meeting. The migration wasn’t news — we’ve been chipping away at it for months (see: the battle plan, the GPU complications, how resolution actually works). But this was the moment it stopped being “something we’re working on” and became “the thing we’re doing instead of the alternative.”
The Side Plots
The meeting surfaced other tensions that are worth noting, because they’re the kind of thing that makes monorepo CI a people problem, not just a tools problem.
Cattle vs. Pets
Our squad lead had the best framing for the flaky test standoff: “For us, these are all cattle. For them, it’s their precious test.”
DevEx wants green CI. We see flaky tests as noise that erodes trust in the entire system. Researchers see their test as the last line of defense for code they spent weeks writing. Neither side is wrong. The incentives just point in opposite directions.
The planned fix is a Datadog test quarantine integration — one of our engineers is building it. Quarantined tests still run and collect data, but they don’t block CI. It’s the “we hear you, but also, your test fails 30% of the time and you need to see that” compromise.
Force Merge Culture
We had a real conversation about when force merging is acceptable. The consensus: impatience is never a valid reason. Force merge exists to unblock main when you have a fix PR that would pass on its own but can’t merge because main is already red from something else. It does not exist because you’re tired of waiting for CI.
The fact that the oncall engineer himself force-merged something — and was genuinely confused about why — is the kind of honesty that makes a team better. The problem isn’t that someone broke a rule. The problem is that the pressure to keep things moving creates a gravitational pull toward “just merge it.”
Pinning Your Tools
Separately, UV’s GitHub Action shipped a broken version that broke our CI. One of our engineers had to pin it. This was the second time an external tool update caused a surprise CI failure. The lesson: if it touches your build, pin it. Let someone else discover the bugs.
Why This Isn’t a Failure Story
find_changed was the right tool when it was built. We had a monorepo growing faster than our CI could keep up. We needed test selection. Bazel wasn’t ready for us yet — not with our GPU dependencies, not with our Python ecosystem complexity. The tool bought us time.
But it hit diminishing returns. Every hour spent improving its import tracing is an hour not spent on Bazel migration. Every new heuristic we add to its change detection is a heuristic Bazel’s sandbox makes unnecessary. We were maintaining two systems that solve the same problem at different levels of rigor.
The decision wasn’t “find_changed is broken.” It was “find_changed has reached the point where continuing to invest in it means investing against the system that replaces it.”
Sometimes the best thing you can do for a tool you built is stop building it.
This is part of an ongoing series on migrating a GPU-heavy Python monorepo to Bazel. Previously: the GPU reality check, the migration battle plan, how Python dependency resolution works in Bazel, and how Bazel’s sandbox breaks Python tests.