Teaching Bazel to See Non-Python Dependencies

The last post had an honesty section. Four things the dep graph doesn’t solve. First on the list:

find_changed’s non-Python deps. Bazel’s dep graph is Python-level. Dockerfiles, C++ sources, cmake configs — find_changed tracks these via package_extra_dependencies.yml. Bazel doesn’t know about them yet.

That was true when I wrote it. It’s not true anymore.

The problem, concretely

Our monorepo isn’t just Python. Some of the most critical packages — the ones that build CUDA kernels, compile C++ extensions, assemble Docker images — depend on files that no Python import scanner will ever see. Dockerfiles. CMakeLists.txt. C++ source trees. .gitmodules. A docker-bake.hcl that orchestrates the whole image build.

find_changed knows about these because someone, at some point, sat down and wrote a YAML file:

mai_kernels: &mai_kernels
  - .gitmodules
  - mai_kernels/CMakeLists.txt
  - mai_kernels/csrc/**

simplertransformer: &simplertransformer
  - *mai_kernels
  - docker-bake.hcl
  - docker/simplertransformer/Dockerfile
  - simplertransformer/CMakeLists.txt
  - simplertransformer/csrc/**

rocket:
  - *simplertransformer
  - .gitmodules
  - docker/Dockerfile
  - docker/rocket/Dockerfile
  - rocket/mt_kernels/**
  - rocket/mango/runit/**

That’s package_extra_dependencies.yml. YAML anchors encode the transitive relationships — simplertransformer inherits everything from mai_kernels, rocket inherits everything from simplertransformer. It’s clever. It works. And Bazel has no idea it exists.

When someone changes docker/Dockerfile, find_changed knows to retest rocket. Bazel’s rdeps query returns nothing, because as far as Bazel is concerned, a Dockerfile is not a dependency of anything. It’s just a file sitting in a directory that Bazel has never heard of.

The insight: Bazel doesn’t need to build things to know about them

Here’s the thing I kept not seeing. I was stuck on: “Bazel can’t build Docker images, so it can’t track Docker dependencies.” Which is true. Bazel’s Python rules don’t know what a Dockerfile is.

But Bazel has a target type that represents arbitrary files: filegroup(). No compilation. No rules. Just: “these files are a thing, and that thing has a name.”

# docker/BUILD.bazel
filegroup(
    name = "pretraining_dockerfiles",
    srcs = [
        "Dockerfile",
        "Dockerfile.experimental",
        "pretraining/Dockerfile",
    ],
    visibility = ["//visibility:public"],
)

That target doesn’t build anything. It doesn’t know what Docker is. It’s a named collection of files. But now //docker:pretraining_dockerfiles is a Bazel target — queryable, dependable, part of the graph.

Wire it as a data dep on a py_library stub, and the edge exists:

# mai_job/BUILD.bazel
py_library(
    name = "mai_job",
    # ... normal deps ...
    data = [
        "//docker:pretraining_dockerfiles",
        "//mai_kernels:native_sources",
    ],
    tags = ["manual"],
)

Now ask Bazel what depends on those Dockerfiles:

$ bazel query 'rdeps(//..., //docker:pretraining_dockerfiles)'
//mai_distributed:mai_distributed
//mai_evaluator:mai_evaluator
//mai_job:mai_job

Three packages. Same three that package_extra_dependencies.yml would tell you. Except now the answer comes from the same graph that handles every other dependency in the repo. One graph. One query language. One source of truth.

Translating the YAML into BUILD files

The YAML has a structure. YAML anchors create a transitive dependency chain:

mai_kernels: &mai_kernels        # base
simplertransformer: &simplertransformer
  - *mai_kernels                 # inherits mai_kernels
rocket:
  - *simplertransformer          # inherits simplertransformer (which includes mai_kernels)

In Bazel, you don’t duplicate the file lists. You express transitivity the way Bazel already understands it — with deps:

# mai_kernels/BUILD.bazel (the native_sources filegroup)
filegroup(
    name = "native_sources",
    srcs = glob(["csrc/**"]) + [
        "CMakeLists.txt",
    ],
    visibility = ["//visibility:public"],
)

# simplertransformer/BUILD.bazel
py_library(
    name = "simplertransformer",
    data = [
        "//docker:simplertransformer_dockerfiles",
        "//mai_kernels:native_sources",
    ],
    deps = ["//mai_kernels"],  # transitive: picks up mai_kernels' data deps
    tags = ["manual"],
)

# rocket/BUILD.bazel
py_library(
    name = "rocket",
    data = [
        "//docker:rocket_dockerfiles",
    ],
    deps = ["//simplertransformer"],  # transitive: picks up simplertransformer + mai_kernels
    tags = ["manual"],
)

The YAML anchor *mai_kernels becomes a Bazel deps edge. The anchor expansion — where simplertransformer’s full file list includes everything from mai_kernels — becomes Bazel’s built-in transitive dependency resolution. Same information. Different encoding. The Bazel encoding is queryable.

The glob() gotcha

First thing I tried: put all the filegroups in a central BUILD file. One place, all the non-Python deps, nice and organized.

Doesn’t work. glob() in Bazel is relative to the BUILD file’s directory. You can’t glob(["mai_kernels/csrc/**"]) from a BUILD file in bazel/tools/. The glob has no idea where mai_kernels/csrc/ is, because it’s looking relative to itself.

So the filegroups live near their files. mai_kernels/BUILD.bazel has the native_sources filegroup for mai_kernels/csrc/**. docker/BUILD.bazel has the Dockerfile filegroups for docker/. Root-level exports_files for things like .gitmodules and docker-bake.hcl that live at the repo root.

This is actually better. The filegroup is co-located with the files it describes. When someone adds a new C++ source under mai_kernels/csrc/, the glob() picks it up automatically. No YAML to update. No manual list to maintain. The glob is the specification.

What changed in the repo

The actual PR touches five things:

1. Root exports_files for repo-root files that packages depend on:

# BUILD.bazel (repo root)
exports_files([
    ".gitmodules",
    "docker-bake.hcl",
])

2. docker/BUILD.bazel with filegroups organized by consumer:

filegroup(
    name = "pretraining_dockerfiles",
    srcs = ["Dockerfile", "Dockerfile.experimental", "pretraining/Dockerfile"],
    visibility = ["//visibility:public"],
)

filegroup(
    name = "simplertransformer_dockerfiles",
    srcs = ["simplertransformer/Dockerfile"],
    visibility = ["//visibility:public"],
)

filegroup(
    name = "rocket_dockerfiles",
    srcs = ["Dockerfile", "rocket/Dockerfile"],
    visibility = ["//visibility:public"],
)

3. Package BUILD files with data deps pointing to the filegroups. Each package that has entries in package_extra_dependencies.yml gets its stub updated with the corresponding data attributes.

4. Seven entries removed from .bazelignore. These directories now have BUILD files (either real ones from prior migration or new stubs), so Bazel can see them. gazelle:exclude directives prevent Gazelle from auto-generating anything — the stubs are intentional, and we don’t want Gazelle “helping.”

5. sync_non_python_deps.py — a validation script that reads package_extra_dependencies.yml, reads the BUILD files, and checks that they agree. Think of it as a drift detector. If someone adds a new entry to the YAML without updating the BUILD file, or vice versa, CI catches it.

The fifth point matters more than it looks. The YAML file is the existing source of truth. The BUILD files are the new encoding. During the transition, they need to agree. After the transition, the BUILD files become the source of truth, and the YAML can be retired.

The tags = ["manual"] contract holds

Same trick as the dep graph post. The stubs are manual-tagged. bazel build //... doesn’t touch them. bazel test //... doesn’t touch them. They exist only for bazel query — graph edges, not build targets.

The contract:

• bazel query //... loads all targets including stubs. Graph is complete.
• bazel build //... skips manual-tagged targets. Existing builds unaffected.
• bazel query 'rdeps(//..., //target)' traverses through stubs. Impact analysis works.

No new build complexity. No new test complexity. Just new graph edges.

Verification: the receipts

The query that proves it works:

$ bazel query 'rdeps(//..., //mai_kernels:native_sources)'
//mai_distributed:mai_distributed
//mai_evaluator:mai_evaluator
//mai_job:mai_job
//mai_kernels:mai_kernels
//rocket:rocket
//simplertransformer:simplertransformer

Change a C++ source file under mai_kernels/csrc/ → Bazel now knows six packages are affected. Before this PR: zero.

$ bazel query 'rdeps(//..., //docker:pretraining_dockerfiles)'
//mai_distributed:mai_distributed
//mai_evaluator:mai_evaluator
//mai_job:mai_job

Change the pretraining Dockerfile → three packages affected. This matches exactly what package_extra_dependencies.yml would tell find_changed. Same answer, different system.

And the builds still pass:

$ bazel build //...   # manual targets invisible — existing builds unaffected
$ bazel test //...    # manual targets invisible — existing tests unaffected

Transitive deps, for free

The YAML uses anchors to encode transitivity. simplertransformer includes *mai_kernels, so if a mai_kernels file changes, simplertransformer is affected too. rocket includes *simplertransformer, so it inherits the whole chain.

In Bazel, this is just… how deps works. rocket depends on //simplertransformer. simplertransformer depends on //mai_kernels. Change a file in mai_kernels:native_sources → Bazel traces through simplertransformer → arrives at rocket. The transitive closure falls out of the graph automatically.

No anchor expansion. No flattening logic. No worrying about whether a deeply nested YAML anchor actually propagates correctly. The graph does transitivity. That’s what graphs are for.

The two-graph convergence

The first post in this series identified six types of dependencies. The CI selector saw some. Bazel saw others. Non-Python deps — type #6, “no import statement, no BUILD rule, just a YAML annotation” — were firmly in find_changed’s territory.

Not anymore. Those relationships are now expressed as Bazel graph edges. The two graphs agree on one more thing than they did yesterday.

Here’s the updated scorecard:

Two rows still favor find_changed. But the one that mattered most for our use case — the non-Python deps that trigger Docker rebuilds and C++ recompilation — is converged.

What this means for CI

Today, when someone changes docker/Dockerfile:

• find_changed checks package_extra_dependencies.yml → retests rocket, mai_job, mai_distributed, mai_evaluator
• Bazel’s rdeps returns nothing, because Bazel doesn’t know Dockerfiles are dependencies

After this PR:

• find_changed does the same thing
• Bazel’s rdeps returns the same packages
• Both systems agree

The practical impact isn’t immediate — we’re not routing CI through Bazel rdeps for non-Python changes yet. But we can. The information is in the graph. When we do build Bazel-powered test selection for the full repo, it won’t have a blind spot for Dockerfiles and C++ sources. The gap that would have made us keep find_changed around forever is closed.

The pattern: graph edges without build rules

This is the generalizable insight, and it’s simpler than I expected.

Bazel doesn’t need to build a thing to know about it. filegroup() targets have no build logic. They’re labels on file collections. Wire them into the graph via data deps, and the graph grows without the build system needing to understand C++, Docker, cmake, or anything else.

The pattern:

1. filegroup() near the non-Python files — lets glob() work
2. data = [...] on the consuming package’s stub — creates the graph edge
3. deps = [...] between stubs — encodes transitive relationships
4. tags = ["manual"] on everything — the graph exists, builds are unaffected

That’s it. Four lines of Starlark per relationship, and Bazel’s query engine handles the rest.

You could apply this to anything. Terraform configs. Protobuf definitions. YAML templates. Kubernetes manifests. Any file that a package depends on but doesn’t import — give it a filegroup, wire it as data, and now the graph knows.

What’s left

From the original four gaps:

Three to go. Version bumps are the hardest conceptually — Bazel’s content-addressable caching means it already re-runs tests when inputs change, but “the lockfile changed” isn’t naturally a Bazel concept. That’s a future post.

For now: Dockerfiles, C++ sources, cmake configs, .gitmodules, docker-bake.hcl — they’re all in the graph. rdeps sees them. One query, one answer, one graph.

The gap that existed for the entire history of our Bazel migration — “but what about non-Python deps?” — is closed. And it only took filegroup().

This is part of an ongoing series on migrating a GPU-heavy Python monorepo to Bazel. Previous posts: Your Monorepo Has Two Dependency Graphs, When Homegrown Tools Hit the Ceiling, The Dep Graph Is the Product, What a Full Dep Graph Actually Unlocks.