Our monorepo has about 90 Python packages. It also has two completely independent systems that answer the question “what depends on what?” They read a lot of the same files. They arrive at different conclusions. Both are correct.
This took me a while to understand. Not the code — the code is readable. What took me a while was accepting that dependency isn’t a single concept. It’s a word we use for at least four different relationships, and no single tool captures all of them.
System 1: The homegrown CI selector
The first system decides which tests to run for a given PR. You changed three files — it figures out which of the 90 packages are affected and builds a test matrix for GitHub Actions.
Under the hood, it maintains two parallel graphs:
A package-level graph built from uv.lock. Every workspace package and external dependency is a node. Edges are dependency relationships, including version tracking. If numpy goes from 1.24 to 1.25 in the lockfile, every package that depends on numpy gets retested.
A file-level graph built from ruff analyze graph plus some Python AST scanning. Every .py file is a node. Edges are imports. This catches changes at finer granularity than package-level — if you touch one file in a package, the system knows which other files import it, and traces the impact upward.
The brain is a DFS traversal on these graphs. You changed foo.py → trace everything that transitively depends on foo.py → those packages get tested.
So far this sounds like a normal import scanner. Here’s where it gets interesting.
The stuff import scanning can’t see
Imports aren’t the only dependencies that matter. The CI selector has escape hatches for everything that Python’s import system doesn’t know about:
Non-Python dependencies. There’s a config file — package_extra_dependencies.yml — that maps packages to Dockerfiles, C++ source files, .gitmodules, CMakeLists.txt. Change a Dockerfile that a package’s build depends on? Retest that package. An import scanner would never see this.
Runtime annotations. Some files have inline TOML comments — # /// [yolo.ci] — that declare runtime dependencies invisible at import time. Config files that get loaded by path, data files, that kind of thing. The CI selector reads these and adds edges to the graph.
conftest.py propagation. In pytest, a conftest.py affects every test in its directory and below. The CI system knows this. If you change a conftest, every test under that directory tree gets flagged.
External version bumps. This is the subtle one. Most CI systems ask “did any source files change?” This one also asks “did any dependency versions change in the lockfile?” A numpy version bump with zero code changes still triggers retesting for every package that uses numpy. Because a new numpy is a new dependency — the behavior might be different even if your code isn’t.
The output: a JSON test matrix. Only the affected packages, sharded across GitHub Actions runners. Change a leaf package that nothing depends on? One shard. Change a core library? Forty shards.
System 2: Bazel + Gazelle
The second system is Bazel, with Gazelle auto-generating BUILD files from Python imports.
Different question, different mechanism. The CI selector asks “what changed?” Bazel asks “what does this target need to build?”
Gazelle scans your Python source files, finds import statements, and maps them to Bazel targets via a 860-line gazelle_python.yaml file (the modules mapping). It generates BUILD files with the correct deps for each target. If you import numpy, Gazelle writes @pypi//numpy in your deps.
The granularity is different too. The CI selector works at package level — change anything in mai_layers, and all of mai_layers’ tests run. Bazel works at target level — change one file, and only the targets that depend on that file rebuild and retest.
And Bazel has its own override mechanisms. # gazelle:resolve tells Gazelle “when you see this import, map it to this target.” # gazelle:ignore tells it “pretend this import doesn’t exist.” These are per-file source annotations that modify the dependency graph, invisible to any other tool.
Currently Bazel covers about 20 of the 90 packages. Migration is in progress. For the other 70, the homegrown system is the only game in town.
Same codebase, different graphs
Here’s where it gets fun. Both systems read the same source files. Both systems reason about dependencies. They produce different graphs.
The canonical example is torch.
Our CI selector scans import torch in a file and adds an edge: this file depends on the torch package. Normal. Correct. The CI system now knows that if torch changes (or if the file changes), things need retesting.
Gazelle scans the same file and sees:
import torch # gazelle:ignore torch
That # gazelle:ignore directive tells Gazelle to drop torch from the dependency graph entirely. Gazelle produces a BUILD file with no torch dep. The import still works at runtime because torch is system-installed in the Docker image — but as far as Bazel’s build graph is concerned, torch doesn’t exist.
Same file. Same import statement. Two systems, two different answers about whether this file depends on torch.
And both are correct for their purposes. The CI selector needs to know: “if torch changes, should we retest this?” Yes. Gazelle needs to know: “can I build this target in Bazel’s sandbox?” Only if I pretend torch isn’t there, because torch isn’t in the Bazel dependency graph on Linux.
What each sees that the other can’t
This isn’t just about torch. Each system has structural blind spots that the other covers.
Things the CI selector sees that Bazel doesn’t:
| Dependency Type | Example | Why Bazel Misses It |
|---|---|---|
| Non-Python files | Dockerfile changes | Bazel’s Python rules don’t track Dockerfiles |
| External version bumps | numpy 1.24 → 1.25 | Bazel caches by target hash, not lockfile diff |
| Runtime file deps | Config loaded by path | Only declared via inline TOML annotations |
| conftest propagation | Shared test fixtures | Bazel treats each test target independently |
Things Bazel sees that the CI selector doesn’t:
| Dependency Type | Example | Why CI Misses It |
|---|---|---|
| Per-target precision | Change one file, rebuild only its dependents | CI operates at package granularity |
| Build-time correctness | Missing dep = immediate build failure | CI only catches this when tests actually run |
| Manual resolve overrides | # gazelle:resolve py foo @custom//bar |
CI’s AST scanner doesn’t read Gazelle directives |
| Hermetic dep tracking | Exact wheel versions, checksummed | CI trusts the lockfile but doesn’t verify artifacts |
The lockfile paradox
Both systems read uv.lock. Same file. Different purposes.
The CI selector uses uv.lock for change detection. It diffs the lockfile between commits to find version bumps. numpy went from 1.24 to 1.25? Find everything that depends on numpy, add it to the test matrix.
Bazel uses uv.lock for hermetic resolution. It reads the lockfile to create @pypi// targets — exact versions, exact wheels, exact checksums. The lockfile isn’t a change signal; it’s the source of truth for what gets installed in the sandbox.
Same input file, different questions. “What changed?” vs. “What should exist?”
This is why it’s wrong to think of these as redundant systems. They share data but not purpose. Merging them would mean one tool trying to answer both questions, and the answer to “what changed?” has almost nothing in common with the answer to “what should the build graph look like?”
The granularity tax
The CI selector operates at package level. You change one file in mai_layers, and every test in mai_layers runs. Maybe 200 tests, when only 3 are actually affected by your change. That’s the tax — false positives. Tests that run unnecessarily.
Bazel operates at target level. Change one file, and only the targets that transitively depend on that file get rebuilt and retested. Maybe 3 tests instead of 200. Much more precise.
But here’s the thing: precision has its own cost. Bazel achieves target-level granularity by requiring every dependency relationship to be explicitly declared in BUILD files. 90 packages, hundreds of files each, thousands of dependency edges — all must be correct, or things either fail to build or silently get the wrong version of a module.
The homegrown system gets away with package-level granularity because it’s cheap. Parse some TOML, scan some ASTs, run DFS. Imprecise but reliable, and fast to build. Bazel’s precision requires a much larger upfront investment — Gazelle, modules mappings, BUILD file generation, override mechanisms — and the ongoing cost of keeping all of it in sync.
Package-level selection with some false positives might run 200 tests when 3 are affected. But it never misses an affected test due to a stale BUILD file. Bazel’s precision is better when it’s correct, but “when it’s correct” is doing a lot of work in that sentence.
Dependencies aren’t one thing
Step back and count how many different things “dependency” means in this codebase:
- Import-time dependency. File A runs
import B. Python’s import system needs B to exist. - Build-time dependency. Target A needs target B compiled first. Bazel’s build graph.
- Version dependency. Package A requires numpy >= 1.24. The lockfile’s constraint solver.
- Change dependency. If file X changes, package Y needs retesting. The CI selector’s change propagation graph.
- Runtime dependency. Package A loads config from a path that happens to be generated by package B’s build step. Invisible to every static tool.
- Non-code dependency. Package A’s tests depend on a Dockerfile that sets up the test environment. No import statement, no BUILD rule, just a YAML annotation that the CI selector reads.
No single tool sees all six. Python’s import system sees #1. Bazel sees #1 and #2. The lockfile captures #3. The CI selector covers #1, #3, #4, and #6. Runtime dependencies (#5) are the darkest corner — you discover them when something breaks in production.
The interesting realization isn’t that our monorepo has two dependency systems. It’s that two isn’t enough. Each system covers a subset of the actual dependency relationships, and the union still has gaps. The question isn’t “which system is right?” — they’re both right about different things. The question is: what does each system’s blind spots cost you?
In our case: the CI selector’s blind spot (package-level granularity) costs us wasted CI minutes — running 200 tests when 3 are affected. Bazel’s blind spot (no torch, no non-Python deps, no version-change detection) costs us coverage — 70 packages that can’t use it yet, and gazelle:ignore directives that create invisible gaps in the build graph.
The convergence
Eventually these two systems should collapse into one. When Bazel covers all 90 packages, the homegrown CI selector becomes a thin layer: “did anything change? yes → run bazel test //... and let Bazel’s caching figure out what actually needs testing.”
But “eventually” is doing heavy lifting. Getting there requires:
-
Solving the system-installed package problem. Torch, flash-attn, triton — they’re provided by Docker, not by the lockfile. Bazel has no native concept of “this Python package is provided by the environment.” We’re working on this now, and it’s the kind of problem that sounds simple (“just tell Bazel where torch is”) and is actually three different problems depending on your platform.
-
Migrating the remaining 70 packages. Each one needs BUILD files, correct dependency declarations, and test targets that actually pass in Bazel’s sandbox. At our current rate, this is months of work.
-
Trusting Bazel’s caching. Right now, running
bazel test //...on every PR sounds terrifying. Bazel’s caching is theoretically perfect — same inputs, same outputs, skip the rest. In practice, cache invalidation in a 90-package monorepo with native deps and GPU builds is… not always perfect. The homegrown system exists partly because “just run everything that might be affected” is a simpler correctness argument than “trust that the cache correctly determined nothing needs to run.”
We’re maybe 20% of the way there. The homegrown system isn’t technical debt — it’s the only system that works for the full codebase today. It’ll become technical debt the day Bazel can handle everything. That day isn’t today.
The general pattern
If you’re running a monorepo above a certain size, you probably have multiple dependency systems too. Maybe it’s your test selector and your build system. Maybe it’s your IDE’s language server and your lockfile resolver. Maybe it’s your Docker build graph and your package dependency graph.
They disagree. They always disagree. And the disagreement isn’t a bug — it’s a consequence of each tool asking a different question about the same code.
The mistake is trying to canonicalize one system as “the” dependency graph. There isn’t one. There are multiple overlapping projections of a higher-dimensional dependency space, and each tool gives you a 2D shadow of it. Some shadows are more useful than others for a given purpose. None of them show you the whole shape.
The practical takeaway: when your CI runs tests that shouldn’t need to run, or when your build system misses a dependency that your test suite catches, don’t blame the tools. Ask what each tool can see and what it can’t. Then decide which blind spots you can live with, and which ones need a YAML file, an inline annotation, or — when all else fails — another system.