The Dep Graph Is the Product: bazel query Over Migration

We had 27 packages on Bazel out of 104. That’s 26% coverage. And for months, the pitch was “migrate more packages” — get BUILD files passing, get tests running, inch the number up. We got it to 54. Good progress. Real work.

But bazel query only sees packages with BUILD files. And if your graph has holes, you can’t ask “what does changing lib/fortknox affect?” because half the answers are invisible. A graph with holes isn’t a graph. It’s a few disconnected islands.

Today I stopped trying to fill in the holes by migrating packages. I filled them by cheating.

The trick nobody talks about

Bazel has a tag called manual. A target with tags = ["manual"] is:

• Visible to bazel query — the dependency graph sees it
• Invisible to bazel build //... and bazel test //... — wildcards skip it

Read that again. You get the graph without any obligation to actually build or test anything. The target is a ghost — it exists for the graph and nothing else.

# AUTO-GENERATED stub for dep graph analysis. DO NOT EDIT.
# gazelle:ignore
load("@aspect_rules_py//py:defs.bzl", "py_library")

py_library(
    name = "mai_config",
    srcs = glob(
        ["**/*.py"],
        allow_empty = True,
    ),
    imports = [".."],
    tags = ["manual"],
    visibility = ["//visibility:public"],
    deps = [
        "//bazel/stubs:torch",
        "@pypi//hydra_core",
        "@pypi//pydantic",
    ],
)

This BUILD file contributes zero to Bazel build/test coverage. But bazel query rdeps(//..., //mai_config/...) now correctly reports every package that depends on mai_config — and transitively, every package that depends on torch.

The dep graph doesn’t care if a target actually builds. It only cares that the edges are declared. So I declared them — for every package in the monorepo.

What I actually built

A generator script that reads pyproject.toml and emits BUILD files with tags = ["manual"]. The whole thing.

The generator

bazel/tools/generate_manual_builds.py does three things:

1. Read a package’s pyproject.toml
2. Classify each dependency into one of three buckets
3. Emit a py_library target with the right labels

The classification is the interesting part:

Workspace deps need their import name, not their package name. mai-config lives at mai_config/src/mai_config — the import is mai_config, not mai-config. The generator discovers import names by listing {pkg}/src/*/ on disk — whatever directories exist under src/ are the import names.

External deps need normalization for @pypi// labels. pyproject.toml might say mai-config or mai_config or Mai-Config — they’re all the same package. Normalize to lowercase, replace hyphens and dots with underscores to match @pypi// label conventions, done.

System stubs

The clever bit. Torch, triton, flash-attn, vllm — these are system-installed in Docker, not in uv.lock. So @pypi//torch doesn’t resolve. And you can’t just omit them — then the graph is missing those edges.

Solution: empty stub targets.

# bazel/stubs/BUILD.bazel
py_library(
    name = "torch",
    tags = ["manual"],
    visibility = ["//visibility:public"],
)

That’s it. A py_library with no srcs, no real deps. A placeholder that exists only so //bazel/stubs:torch resolves. Fourteen of these for the system-installed packages: torch, triton, flash_attn, vllm, sglang, xgrammar, and friends.

The stub says: “torch exists. I am not it. But if you need to draw an edge to torch, point it here.” The graph gets the topology right even though the target has no substance.

Gazelle doesn’t know about the stubs

Here’s a gap I didn’t notice until someone asked the obvious question: what happens when a fully-migrated package imports torch?

For manual stubs, # gazelle:ignore means Gazelle never touches the BUILD file. We write //bazel/stubs:torch by hand, done. But for Gazelle-managed packages — the ones we actually migrated — Gazelle is in charge of the deps list. And Gazelle has no idea what import torch maps to.

Torch isn’t in gazelle_python.yaml (the manifest that maps Python imports to Bazel targets). There’s no gazelle:resolve directive for it anywhere in the repo. So when Gazelle encounters import torch, it shrugs. The developer either annotates the import with # gazelle:ignore torch or manually adds //bazel/stubs:torch to the BUILD file and hopes the next Gazelle run doesn’t clobber it.

The evidence is scattered across the codebase: only 3 files have # gazelle:ignore torch annotations. 15+ BUILD files have manually added //bazel/stubs:torch deps. Zero gazelle:resolve directives for torch exist. Everyone reinvents the workaround independently.

The fix is one line in the root BUILD.bazel:

# gazelle:resolve py torch //bazel/stubs:torch

That tells Gazelle: “when you see import torch, resolve it to the stub.” Works globally for every package. Same trick works for all 14 system stubs — triton, flash_attn, vllm, the whole set.

One nice thing falls out of this for free: transitive deps just work. If mai_config depends on //bazel/stubs:torch and mai_cluster depends on mai_config, Bazel propagates the torch edge through the graph automatically. mai_cluster doesn’t need its own stub dep. The py_venv_test with include_system_site_packages=True handles runtime availability — the graph just needs the edges, and transitivity gives them.

The .bazelignore problem

Here’s where it got hairy.

Bazel has a .bazelignore file — directories listed there are completely invisible. Not just “not built.” Invisible. Can’t query them. Can’t reference them. They don’t exist.

We had ~70 packages in .bazelignore. Every unmigrated package was ignored, because if Bazel wandered into those directories and found no BUILD file, it would error. Reasonable at the time.

But now I’m adding BUILD files to those directories. Which means they need to come OUT of .bazelignore. All ~70 of them.

This is where you hold your breath and see if anything breaks. (Spoiler: the existing builds still pass. Because tags = ["manual"] means these new targets are invisible to bazel build //.... The graph changed. The build didn’t.)

Intermediate BUILD files

One gotcha: directories that were in .bazelignore might not have BUILD files at any level. Once you remove them from .bazelignore, Bazel can wander into them. The monorepo has directories like agents/, apps/, lib/, posttraining/ that contain packages but aren’t packages themselves. Adding an empty BUILD file to each makes them proper Bazel packages — clean, explicit, no surprises:

# agents/BUILD.bazel
# Intermediate BUILD file for Bazel package resolution.

Literally empty. Just enough to make the directory a proper Bazel package.

Edge cases that bit me

Every monorepo script has a graveyard of edge cases. Here are the ones that drew blood:

codeshield — most packages have their source at {pkg}/src/{import_name}/. codeshield has it at lib/codeshield/src/codeshield/. The BUILD file needs to go at the import root, not the package root. The generator handles this by actually looking at what’s on disk rather than assuming the layout.

Self-referential deps — a package’s pyproject.toml sometimes lists itself. research_utils depends on research-utils. That’s a cycle of length 1. The generator filters these out.

Packages without pyproject.toml — in a monorepo, each package directory has its own pyproject.toml declaring its dependencies. Some directories look like packages — they’re sitting in lib/ or apps/ — but have no pyproject.toml, no source. The generator skips them and moves on.

# gazelle:ignore — every generated BUILD file gets this directive at the top. Without it, Gazelle (Bazel’s auto-generator for fully migrated packages) would overwrite the manual stubs with whatever it thinks the BUILD file should be. The stubs are intentionally incomplete — Gazelle would “fix” them by adding deps it can’t resolve, breaking everything.

The numbers

Before and after, same repo, same code:

Some queries I couldn’t run before:

# "What depends on mai_config?"
$ bazel query 'rdeps(//..., //mai_config/...)' | wc -l
65

# "What depends on mai_nano?"
$ bazel query 'rdeps(//..., //lib/mai_nano/...)' | wc -l
319

# "What depends on fortknox?"
$ bazel query 'rdeps(//..., //lib/fortknox/...)' | wc -l
60

# "How many test targets exist?"
$ bazel query 'kind(py_test, //...)' | wc -l
166

That rdeps on fortknox is the one that matters for the pitch. Change fortknox → 60 out of 986 targets are affected → 94% of targets can be skipped. That’s the headline number.

And the existing builds? Still green:

$ bazel build //ci/... //lib/mai_nano/...
# ✅ passes — manual targets are invisible to build wildcards

Proving it works

People hear “manual stubs” and “generated BUILD files” and their brain goes: “okay but does any of this actually work?” Fair. Here’s exactly what needs to pass, what we tested, and why the bar for success is lower than you think.

The key insight

The goal is a complete dependency graph queryable via bazel query rdeps(). Not building. Not testing. The graph. So the validation question isn’t “does everything build?” — it’s “does the graph load, and can you traverse it?”

Think of it like a city map. You don’t need every building to be structurally sound to draw the map. You just need the addresses and connections to be correct. The stubs are pins on a map — they mark “this package exists, and it connects to these other packages.” That’s enough for routing.

What MUST pass (CI-gated)

These run on every PR. If any fail, the graph is broken.

1. bazel query //... loads without errors. This is the foundational check. If even one BUILD file has a typo, missing dep label, or bad glob, this fails. Every target in the workspace must parse. We loaded 986 targets across ~290 BUILD files.

2. bazel build //... still passes. Proves the manual stubs don’t break existing real builds. The tags = ["manual"] should exclude stubs from build wildcards — that’s the whole contract. 1174 actions, all pass.

3. bazel test //... --test_tag_filters=-integration,-torch still passes. Same argument. The stubs are invisible to test wildcards. 121/121 tests pass, 1858/2051 test cases pass.

What MUST work (proven locally)

These aren’t CI-gated yet, but they’re the actual proof of the goal.

4. rdeps() queries resolve across packages. This is THE validation. If you can query “what depends on mai_config” and get 65 packages back, the graph works. Not “a graph works.” The graph — the one that covers the entire monorepo.

5. Cross-package dep chains resolve. rocket → mai_config → torch stub. The chain traverses workspace deps, through to system stubs. This is what makes targeted test selection possible — the query engine can follow edges across package boundaries, through real targets and stubs alike.

What does NOT need to pass

This is where people’s intuition is wrong.

6. The stubs don’t need to build. They’re manual tagged. bazel build //rocket/src/rocket would fail because the stub is incomplete — it has the dep edges for the graph, but it’s not a fully wired-up build target. That’s fine. We never intend to build these. The BUILD file exists only for bazel query to read its deps = [...] attribute.

7. @pypi// labels resolve via the hub, not at build time. External deps like @pypi//pydantic are registered through uv.lock and resolve during bazel query. But the manual stubs never build those deps — the targets are manual tagged, so the resolved labels are just edges in the graph. No fetching, no compilation.

The receipts

The first three run in CI every PR. The last four are the “graph works” proof — we ran them locally, and the CI annotation step will exercise rdeps() on every PR going forward.

The bar for success was lower than expected. You don’t need builds to pass. You need the graph to load. And it does.

What this actually enables

This was never about BUILD files. It was always about what you can ask once the graph is complete.

Targeted test selection. Change mai_config? Run tests in the 65 packages that depend on it. Not all 166 test targets. The current system (find_changed) does package-level DFS through pyproject.toml deps. Bazel does target-level precision. The difference is “run all tests in every affected package” vs “run the specific tests whose transitive deps changed.”

Scoped pyright. The merge queue currently runs pyright on the entire repo. 25 minutes. With the dep graph, scope it to affected packages. Leaf package change? 2-3 minutes. Core lib change? 8-12 minutes. Still better than 25.

GPU test targeting. GPU runners are expensive. Don’t spin one up because someone edited a README. The graph knows which changes touch GPU-tagged test targets and which don’t.

Impact analysis in PRs. I added a CI annotation step — informational, continue-on-error: true — that shows affected targets in the PR summary. “This change affects 60/986 targets, ~94% skip rate.” Before you merge, you know the blast radius.

Migration planning. This is the recursive one. The graph tells you which packages are ready for full Bazel migration — all their deps are already migrated. No spreadsheet. Just a query.

The superset property

One thing I worried about: the dep graph isn’t the same across environments. On Linux with GPU, mai_kernels imports CUDA extensions. On macOS, those imports don’t exist. Three different runtime dependency graphs.

The BUILD file graph is a superset. It declares every dependency that exists in any environment. If package A depends on B only on Linux GPU, the BUILD file still says deps = ["//B"].

That means bazel query rdeps() might occasionally over-include — telling you to run a test that would actually be skipped at runtime. But it will never under-include.

Over-inclusion is fine. Under-inclusion breaks CI. The superset is a feature.

What could go wrong

I’d be lying if I said this was clean.

BUILD file drift. When someone adds a dependency to pyproject.toml, the generated BUILD file is now stale. For fully-migrated packages, Gazelle handles this. For manual stubs, nobody does. We need a CI check: “does the BUILD file match pyproject.toml?” Run the generator, diff the output, fail if they diverge. Without this, the graph silently rots.

Circular dependencies. Python tolerates circular imports at runtime. Bazel does not tolerate circular deps. If A deps B and B deps A, you can’t express it. A few of these exist in the monorepo. The generator currently handles them by… not handling them. They’ll need manual resolution.

lib/bus and the C++ chain. The stubs work for Python-level deps. But lib/bus is C++/Cython — its real deps can’t be captured in a py_library target. The stub captures its Python-level deps (what its consumers import), not its build-level deps. Incomplete, but better than absent.

Gazelle conflicts. If someone runs Gazelle on a package that has a manual stub, Gazelle will try to regenerate the BUILD file. The # gazelle:ignore directive prevents this, but only if Gazelle respects it. Gazelle does respect it, but it’s one more thing to know.

The metric shift

Here’s what I think the team should be tracking:

A package with tags = ["manual"] scores zero on the old metrics and 100% on the new ones. That’s not a contradiction — it’s a reframe. The old metrics measure migration progress. The new metrics measure what the migration is for.

The punchline

We spent months migrating packages to Bazel. Good work. Necessary work. But the value was never “now we build with Bazel.” The value is “now Bazel knows our dependency graph, and that graph makes CI smart.”

A generator script, 14 stub targets, and removing 70 lines from .bazelignore — that’s what it took to go from 26% graph coverage to ~100%. The manual-tagged targets cost nothing to run (they don’t run), break nothing (they’re invisible to build //...), and are cheap to maintain (regenerate from pyproject.toml).

The full graph is the product. Everything else is scaffolding.

Stop counting packages on Bazel. Start counting edges in the graph.

Previous posts in this series: How Bazel Resolves Python Deps (and Why torch Breaks Everything), Dependency Graph Cascade Analysis, Your Monorepo Is a Graph. Those covered the theory and the analysis. This one covers what happened when we actually built it.