Bazel Migration: How I Gave the Monorepo a Complete Dep Graph

Our monorepo has about 101 Python packages. Until last week, Bazel could see 24 of them. The other 77 were invisible — explicitly hidden, in fact, via a file called .bazelignore. You couldn’t query them. You couldn’t trace their dependencies. If someone asked “what breaks if I change mai_config?”, the honest answer was “we literally cannot tell you.”

This post is how we went from 24% graph coverage to ~100% in about a day, without changing a single build or breaking a single test. If you’re new to Bazel, or your team is migrating a Python monorepo, or you just want to understand what a “dependency graph” actually is and why someone would spend a week obsessing over one — this is the walkthrough.

Some vocabulary, before we start

If you already know Bazel, skip this. If you don’t, here’s the minimum viable glossary. I’ll explain more as we go, but these terms will keep appearing:

You don’t need to memorize these. They’ll make sense in context.

The problem: a graph with holes

A dependency graph is just a map of “what depends on what.” Package A imports package B. Package B imports package C. Draw arrows between them — that’s the graph.

Bazel builds this graph from BUILD files. No BUILD file, no node in the graph. And we had 77 packages with no BUILD file. So our graph had 77 holes in it.

Why does this matter? Because the whole point of a dependency graph is answering questions like:

• “If I change package X, what tests might break?” (rdeps())
• “What’s the most critical package in the repo?” (highest fan-in)
• “What order should I migrate packages in?” (topological sort)
• “Can I safely delete this package?” (zero reverse deps)

A graph with holes can’t answer these. If mai_config depends on torch and 40 other packages depend on mai_config, but mai_config isn’t in the graph — all those edges are missing. Your map has a blank spot right in the middle of the city.

The 24 packages we did have in Bazel? They formed disconnected islands. You could query within an island, but you couldn’t trace across the gaps.

The insight that changed everything

Here’s the thing I wish someone had told me six months ago.

You don’t need to fully migrate a package to get it in the graph. You just need Bazel to know it exists and know what it depends on. The package doesn’t need to build. It doesn’t need to test. It just needs a BUILD file with the right deps list and a magic tag.

That tag is tags = ["manual"].

A target with tags = ["manual"] is:

• Visible to bazel query (the dependency graph sees it)
• Invisible to bazel build //... (wildcards skip it)
• Invisible to bazel test //... (wildcards skip it)

Read that again. You get the graph without any obligation to actually build or test anything. The target is a ghost in the machine — it exists so the graph can see it, and for no other reason.

This means you can create “stub” BUILD files — BUILD files that describe a package’s dependencies for graph analysis, but that never compile anything, never download anything, never run anything. They’re metadata. A card in the catalog for a book that hasn’t arrived yet.

Step 1: Let Bazel see the directories

Before we can add BUILD files, Bazel needs to be allowed to look in those directories.

The .bazelignore file is a list of directories Bazel pretends don’t exist. Not “won’t build.” Don’t exist. Can’t query them, can’t reference them, can’t see them. It’s the nuclear option for telling Bazel to stay out.

Our .bazelignore had about 80 entries — every unmigrated package directory. This made sense when we had no BUILD files for those packages: Bazel wandering into a directory with no BUILD file would error. So we locked the doors.

Now we’re putting BUILD files in those directories. So the doors need to open.

Before (simplified — imagine 60+ more like this):

agents
apps/dataplatform/api
apps/dsviewer
apps/pablo
caas
caas_server
mai_config
mai_trainer
rocket
...

After (only non-Python stuff stays ignored):

# Standard ignores
.venv
.git
.pytest_cache

# Non-Python directories — frontends, infra, scripts
apps/dataplatform/app_service_frontend
apps/dataplatform/frontend
docker
docs
typings
user_scripts
yaml_store

The rule: if it’s a Python package, remove it from .bazelignore. If it’s infrastructure, docs, or frontend code that Bazel has no business looking at, keep it.

This is the step where you hold your breath. Removing 60+ entries from .bazelignore means Bazel will now wander into all those directories. If anything breaks…

Nothing breaks. Because we haven’t added BUILD files yet. And a directory with no BUILD file that’s not in .bazelignore? Bazel just ignores it naturally — it only cares about directories with BUILD files. The .bazelignore was belt-and-suspenders. Removing it changes nothing… until we add the stubs.

Step 2: Handle the Gazelle excludes

Quick sidebar on Gazelle, because this’ll confuse you later if I don’t explain it now.

Gazelle is a tool that auto-generates BUILD files from your source code. You run bazel run //bazel:gazelle, it scans your Python files, figures out what imports what, and writes BUILD files for you. It’s how the 24 already-migrated packages got their BUILD files.

The root BUILD.bazel file had # gazelle:exclude directives for every unmigrated package — telling Gazelle “don’t scan these directories.” We need to update this too, because our new stub BUILD files need a different directive.

Each stub BUILD file gets # gazelle:ignore at the top. This tells Gazelle: “this BUILD file exists, but I’m managing it, not you. Don’t touch it.” Different from # gazelle:exclude (which says “pretend this directory doesn’t exist”). With # gazelle:ignore, the directory exists, the BUILD file exists, Gazelle just won’t overwrite it.

So we replace the per-package excludes in the root BUILD with generic directory excludes (docs, docker, etc.) and let each stub protect itself.

Step 3: Create system dependency stubs

Here’s a fun problem. Some packages depend on torch, triton, vllm, flash_attn — big GPU libraries that are system-installed in Docker, not downloaded from PyPI. So @pypi//torch doesn’t exist in Bazel’s world. There’s no wheel for it. There’s no lockfile entry.

But our BUILD stubs need to say “this package depends on torch.” The dep edge needs to go somewhere. If we just omit torch from the deps list, the graph is wrong — it’s missing real dependency edges.

Solution: create empty placeholder targets.

# bazel/stubs/BUILD.bazel
load("@aspect_rules_py//py:defs.bzl", "py_library")

[py_library(
    name = name,
    tags = ["manual"],
    visibility = ["//visibility:public"],
) for name in [
    "cuda_bindings",
    "flash_attn",
    "torch",
    "torchvision",
    "triton",
    "vllm",
    "sglang",
    # ... 14 total
]]

That’s a list comprehension in Starlark (Bazel’s build language — it’s basically Python). For each system package, we create an empty py_library with no source files, no real dependencies. It’s a pin on the map. A signpost that says “torch exists here” so that other BUILD files can point their deps at it.

Now when a stub BUILD file says deps = ["//bazel/stubs:torch"], that label resolves. The graph gets the edge. Nobody tries to build torch. Everyone’s happy.

Step 4: The generator script

This is the core of the whole thing. A Python script that reads every package’s pyproject.toml and generates a stub BUILD file.

The script lives at bazel/tools/generate_manual_builds.py. Here’s what it does, step by step:

4a: Find all packages

The root pyproject.toml has a [tool.uv.workspace] section listing every workspace member:

[tool.uv.workspace]
members = [
    "agents",
    "apps/dataplatform/api",
    "caas",
    "lib/mai_nano",
    "mai_config",
    "rocket",
    # ... ~100 more
]

The script reads this list. That’s the universe of packages.

4b: Read each package’s dependencies

Each package has its own pyproject.toml with a [project] section:

# mai_config/pyproject.toml
[project]
name = "mai-config"
dependencies = [
    "frozendict",
    "hydra-core>=1.3",
    "numpy",
    "omegaconf",
    "pydantic>=2.0",
    "torch",
]

[tool.uv.sources]
mai-nano = { workspace = true }

Two things matter here: the dependencies list (what this package needs) and tool.uv.sources (which of those deps are other monorepo packages vs. external PyPI packages).

4c: Classify each dependency

Every dependency falls into one of three buckets:

The tricky part is workspace deps. The pyproject.toml says the package name is mai-config, but Python imports it as mai_config. And the Bazel label needs the filesystem path to the import directory. So the script does:

1. Look at {package_dir}/src/ on disk
2. Find what subdirectories exist (e.g., mai_config/src/mai_config/)
3. That subdirectory name is the import name
4. The Bazel label is //mai_config/src/mai_config

For external deps, normalize the name: lowercase, replace hyphens and dots with underscores. hydra-core becomes hydra_core. azure.storage.blob becomes azure_storage_blob. This matches @pypi// label conventions.

4d: Skip already-migrated packages

If a package already has a BUILD file that doesn’t have # AUTO-GENERATED stub in it, it’s fully migrated. Don’t touch it. The generator only creates stubs for packages that aren’t on Bazel yet.

4e: Generate the BUILD file

For each unmigrated package, emit something like:

# AUTO-GENERATED stub for dep graph analysis. DO NOT EDIT.
# gazelle:ignore
load("@aspect_rules_py//py:defs.bzl", "py_library")

py_library(
    name = "mai_config",
    srcs = glob(["**/*.py"], allow_empty = True),
    imports = [".."],
    tags = ["manual"],
    visibility = ["//visibility:public"],
    deps = [
        # system stubs
        "//bazel/stubs:torch",
        # external deps
        "@pypi//frozendict",
        "@pypi//hydra_core",
        "@pypi//numpy",
        "@pypi//omegaconf",
        "@pypi//pydantic",
    ],
)

Let’s unpack this line by line, because each one matters:

• # AUTO-GENERATED stub — so you know not to edit it by hand, and so the generator knows to overwrite it on re-run
• # gazelle:ignore — tells Gazelle to leave this file alone
• load(...) — imports the py_library rule from rules_py
• name = "mai_config" — the target name, matching the import name
• srcs = glob(["**/*.py"], allow_empty = True) — grabs all Python files. Not precise (real BUILD files list files explicitly), but good enough for graph analysis. allow_empty = True means it won’t fail if there are no .py files
• imports = [".."] — tells Python where the import root is (one directory up from src/mai_config/)
• tags = ["manual"] — the magic tag. Graph: yes. Build: no
• visibility = ["//visibility:public"] — any other target can depend on this
• deps = [...] — the actual dependency edges. This is what makes the graph work

4f: Generate intermediate BUILD files

Here’s a gotcha. Some packages live in nested directories: apps/dataplatform/api, lib/mai_nano, posttraining/rlvr. Once these are out of .bazelignore, Bazel needs every parent directory to be a valid Bazel package. That means apps/ needs a BUILD file. apps/dataplatform/ needs a BUILD file. lib/ needs a BUILD file.

These intermediate BUILD files are empty:

# Intermediate BUILD file for Bazel package resolution.

That’s it. One comment. Just enough to make the directory a proper Bazel package so Bazel can traverse through it to find the actual packages inside.

Running it

# Dry run — shows what would be generated, writes nothing
uv run python bazel/tools/generate_manual_builds.py

# Actually write the files
uv run python bazel/tools/generate_manual_builds.py --write

Always dry-run first. The script prints a summary: how many packages processed, how many BUILD files generated, how many skipped (already migrated).

Step 5: Format everything

Bazel has its own formatter called buildifier (like prettier for BUILD files). And our CI checks that all BUILD files are formatted. So after generating 70+ BUILD files, run the formatter:

# Format BUILD files
bazel run //bazel:format

# Format the generator script itself
uv run ruff format bazel/tools/generate_manual_builds.py
uv run ruff check --fix bazel/tools/generate_manual_builds.py

Boring but necessary. If you skip this, CI will yell at you about formatting before it ever gets to the interesting stuff.

Step 6: Verify nothing broke

This is the moment of truth. We just:

1. Removed 60+ entries from .bazelignore
2. Created 70+ new BUILD files with manual-tagged targets
3. Created 14 system dependency stubs
4. Generated intermediate BUILD files for parent directories

Did we break anything?

# Can Bazel see all the targets?
$ bazel query //... 2>/dev/null | wc -l
986

# Do the existing builds still work?
$ bazel build //...
# ✅ All green. Manual targets are invisible to build wildcards.

# Do the existing tests still pass?
$ bazel test //... --test_tag_filters=-integration,-torch
# ✅ 121/121 tests pass.

Three checks. All green. The stubs are ghosts — they exist in the graph, but build //... and test //... can’t see them. That’s the whole contract of tags = ["manual"], and it holds.

Now the interesting check — does the graph actually work?

# "What depends on mai_config?"
$ bazel query 'rdeps(//..., //mai_config/src/mai_config)' | wc -l
65

# "What depends on mai_nano?" (a core utility lib)
$ bazel query 'rdeps(//..., //lib/mai_nano/src/mai_nano)' | wc -l
319

# "How many total test targets exist?"
$ bazel query 'kind(py_test, //...)' | wc -l
166

319 packages depend on mai_nano. Yesterday, Bazel couldn’t tell us that. Today it can. That’s the whole point.

How stubs differ from real BUILD files

It’s worth seeing the difference side by side, because understanding this is key to knowing what “full migration” actually means.

Stub BUILD file (auto-generated, graph-only):

# AUTO-GENERATED stub for dep graph analysis. DO NOT EDIT.
# gazelle:ignore
load("@aspect_rules_py//py:defs.bzl", "py_library")

py_library(
    name = "mai_config",
    srcs = glob(["**/*.py"], allow_empty = True),
    imports = [".."],
    tags = ["manual"],
    visibility = ["//visibility:public"],
    deps = [
        "//bazel/stubs:torch",
        "@pypi//frozendict",
        "@pypi//hydra_core",
    ],
)

Real BUILD file (fully migrated, actually builds):

load("@aspect_rules_py//py:defs.bzl", "py_library")

py_library(
    name = "mai_nano",
    srcs = [
        "__init__.py",
        "enums.py",
        "import_class.py",
        # ... every file listed explicitly
    ],
    imports = [".."],
    visibility = ["//visibility:public"],
    deps = [
        "@pypi//frozendict",
    ],
)

The migration path for any package: delete the stub BUILD file, run Gazelle, fix any issues, remove tags = ["manual"], add test targets. The stub is the “before” picture. The real BUILD is the “after.”

Edge cases that bit me

Every monorepo script has a graveyard of edge cases. Here’s what drew blood:

codeshield — most packages live at {package}/src/{import_name}/. codeshield lives at lib/codeshield/src/codeshield/. The generator needs to find the import root on disk, not assume the layout. It does this by listing {pkg}/src/*/ and seeing what exists.

Self-referential deps — some packages list themselves as a dependency in pyproject.toml. research_utils depends on research-utils. That’s a circular dependency of length 1. The generator filters these out.

Packages without pyproject.toml — some directories in lib/ or apps/ look like packages but have no pyproject.toml. No config, no source. The generator skips them.

Namespace conflicts — two packages might have the same import name. The generator deduplicates and warns.

Optional/extra dependencies — pyproject.toml can have [project.optional-dependencies]. The generator only reads [project.dependencies] — the core deps. Optional deps are a can of worms for another day.

What we got

The before and after:

Some queries that were impossible before:

# "What depends on mai_config?" → 65 packages
$ bazel query 'rdeps(//..., //mai_config/src/mai_config)' | wc -l
65

# "What depends on mai_nano?" → 319 packages (a core hub!)
$ bazel query 'rdeps(//..., //lib/mai_nano/src/mai_nano)' | wc -l
319

# "How many test targets are affected by a fortknox change?"
$ bazel query "kind('py_test', rdeps(//..., //lib/fortknox/src/fortknox))" | wc -l
60
# 60 out of 986 total targets → 94% skip rate

That last one is the headline number for CI. Change fortknox → only 60 targets need testing → skip the other 926. That’s the value of a complete graph.

Bonus: the affected targets CI step

With the graph in place, we added a step to the Bazel CI workflow that runs on every PR. It:

1. Detects which packages changed in the PR
2. For each changed package, queries bazel query "kind('py_test', rdeps(//..., $pkg))"
3. Outputs a table to the GitHub PR summary

It’s purely informational — continue-on-error: true, never blocks a merge. But now every PR has a line like:

This change affects 60/986 targets (~94% skip rate)

Before you review the code, you know the blast radius. Context before code.

What this enables (and what it doesn’t)

What you can do now:

• “What breaks if I change X?” — bazel query 'rdeps(//..., //X/...)'
• Targeted test selection — run only the tests that transitively depend on your change
• Scoped pyright — type-check only affected packages, not the whole repo
• GPU test targeting — only spin up a GPU runner if your change touches torch-tagged targets
• Migration planning — the graph tells you which packages are ready for full migration (all their deps are already migrated)
• Impact analysis — hub detection (what has the most reverse deps?), leaf detection (what has zero?), critical path analysis

What this does NOT do:

• Build the stub packages. Stubs are graph-only. bazel build //rocket/src/rocket on a stub would fail. That requires full migration.
• Run tests for stub packages. Same deal. The stub has no py_test targets. Tests still run through the old system (pytest + find_changed).
• Capture non-Python deps. Dockerfiles, C++ sources, cmake configs — the stubs only capture Python-level deps from pyproject.toml. The existing find_changed system still catches those.
• Detect version bumps. If numpy goes from 1.24 to 1.25, the stubs don’t know. @pypi//numpy is @pypi//numpy regardless of version. find_changed handles this.

The stubs give you the topology — the shape of who-depends-on-whom. They don’t give you hermetic builds, sandboxed tests, or cache-aware execution. Those require real BUILD files.

Think of it this way: the stubs drew the map. Full migration builds the roads.

If you need to add a new package

New package shows up in the monorepo? Here’s the flow:

1. Someone adds a new directory with a pyproject.toml and lists it in the root workspace members
2. Re-run the generator: uv run python bazel/tools/generate_manual_builds.py --write
3. Format: bazel run //bazel:format
4. Commit the generated BUILD file

That’s it. The generator reads the workspace config, finds the new package, generates a stub, and the graph automatically includes it.

Eventually a CI check should enforce this — “does every workspace member have a BUILD file?” — so drift gets caught automatically. But for now, re-running the generator is cheap and fast.

If you need to migrate your package for real

The stub is your starting point. Here’s the upgrade path:

1. Delete the generated stub — remove the # AUTO-GENERATED BUILD file
2. Run Gazelle — bazel run //bazel:gazelle regenerates the BUILD file from actual imports
3. Fix any Gazelle issues — missing gazelle:resolve directives, import mapping problems
4. Remove tags = ["manual"] — your package is now a real build target
5. Add test targets — py_test for your test files
6. Verify — bazel build //your/package/... should work. bazel test //your/package/... should pass.

The stub gave you the graph edge. Migration gives you the build. Different things, both valuable.

The punchline

We spent months migrating packages one at a time. Good work. Necessary work. But the insight was: you don’t need to migrate a package to get it in the graph. You just need to describe it.

A pyproject.toml already describes every package’s dependencies. A script that translates those descriptions into BUILD files with tags = ["manual"] — that’s all it took. Remove some lines from .bazelignore, create 14 system stubs, run a generator, format the output.

From 24% graph coverage to ~100%. Zero broken builds. Zero broken tests. 986 targets visible. 319 reverse deps on mai_nano. 65 on mai_config. Answers to questions we couldn’t even ask before.

The full migration is still happening — stubs need to become real BUILD files, one package at a time. But the graph is done. And a complete graph is the foundation for everything else: smarter CI, scoped type checking, targeted GPU testing, migration planning. You build on the graph.

Stop counting packages on Bazel. Start counting edges in the graph.

This is part of a series on Bazel dep graph work: Your Monorepo Has Two Dependency Graphs (and They Disagree) (the two-graph problem), The Dep Graph Is the Product (the overview), What a Full Dep Graph Actually Unlocks (what it enables). This one is the step-by-step walkthrough — how it actually works, from zero to 100%.