Bazel Migration: 6 Packages, 10 Bumps, 9 Pushes

TL;DR: I migrated 6 Python packages with torch/CUDA dependencies to Bazel in our monorepo. It took 9 CI pushes to get green. Every single push taught me something new about the gap between “Bazel wants hermeticity” and “torch ships a 2GB wheel with CUDA bindings.” This is the full war story — every bump, every fix, every moment of staring at a CI log thinking “that can’t be right.”

This is the fourth post in a series. Part 1 was observational. Part 2 was the battle plan. Part 3 explained the resolution machinery. This post is the execution — the part where plans meet reality and reality wins every round.

The Starting Point

Six packages needed to move to Bazel:

The toolchain: aspect_rules_py for Bazel Python rules, Gazelle for auto-generating BUILD files, uv for package management. The CI: GitHub Actions with both non-Docker (pure Python) and Docker (torch/CUDA) test jobs.

Simple, right? Remove packages from exclusion lists, run Gazelle, push. Maybe a couple iterations.

Nine pushes. Ten distinct problems. Let’s go.

The Three Exclusion Layers

Before any package can be “Bazel-ified,” you have to remove it from three separate exclusion lists. Miss one and you get confusing failures where Bazel seems to half-know about your package.

Layer 1: .bazelignore — Bazel itself skips these directories entirely. They don’t exist as far as the build system is concerned.

Layer 2: # gazelle:exclude in root BUILD.bazel — Gazelle skips these during BUILD file generation. Bazel knows about the directory, but Gazelle won’t auto-generate targets for it.

Layer 3: GAZELLE_DIRS whitelist in bazel/BUILD.bazel — Only explicitly listed directories get Gazelle treatment. A positive allowlist on top of two negative blocklists.

Three layers of “no” that all have to become “yes.” I wrote this down because I got bitten by forgetting Layer 3 on the first package and spent 20 minutes wondering why Gazelle was silently doing nothing.

Bump #1: Gazelle’s # gazelle:ignore Is Pickier Than You Think

Gazelle auto-generates BUILD files by scanning Python imports. For system-installed packages like torch that aren’t in @pypi//, you need to tell Gazelle “don’t try to resolve this.” The directive is # gazelle:ignore <module>.

It works on this:

import torch  # gazelle:ignore torch

It does not work on this:

from torch.nn import (  # gazelle:ignore torch
    Linear,
    Module,
)

Multi-line imports? Gazelle ignores your ignore. It’ll still try to resolve torch, fail, and either emit a broken dependency or silently drop it.

There’s a second gotcha: # gazelle:ignore can’t share a line with other comments. This fails:

import torch  # type: ignore  # gazelle:ignore torch

The fix is mechanical but tedious: rewrite every torch/torchaudio/hydra import as single-line format with # gazelle:ignore as the only comment. Not glamorous. Gets the job done.

Bump #2: ruff Undoes Your Gazelle Workarounds

Here’s where it gets fun. You’ve carefully formatted your imports as single-line with # gazelle:ignore comments. You push. ruff reformats them back to multi-line. Gazelle breaks again.

The specific case:

from datadog.dogstatsd.base import DogStatsd  # gazelle:ignore datadog.dogstatsd.base,datadog.dogstatsd,datadog

That line is long. ruff’s line-length formatter wants to split it. And once it’s multi-line, the # gazelle:ignore comment lands on the first line of the from X import ( block, where Gazelle can’t parse it.

The naive fix: # isort: off to suppress import sorting.

The reality: # isort: off only suppresses ruff’s I001 import-sorting rule. It does NOT suppress line-length formatting. Your line still gets split.

The actual fix: You need both formatters suppressed:

# isort: off
# fmt: off
from datadog.dogstatsd.base import DogStatsd  # gazelle:ignore datadog.dogstatsd.base,datadog.dogstatsd,datadog
# fmt: on
# isort: on

Four comment lines to protect one import. It looks absurd. It’s correct. The alternative is Gazelle generating broken BUILD files on every regeneration, which is worse.

Bump #3: Gazelle Eats Your Hand-Edited Tags

Our CI uses Bazel’s --test_tag_filters to split test execution:

• Non-Docker job: --test_tag_filters=-integration,-torch — skip anything needing GPU or infrastructure
• Docker job: --test_tag_filters=torch,-integration — only run torch tests, skip integration tests

So torch-dependent tests need tags = ["torch"] in their BUILD targets. Easy — just add the tag to the generated py_test target.

Except Gazelle regenerates BUILD files. Every time it runs, your hand-edited tags attribute disappears. You’re playing whack-a-mole with a code generator.

The fix: Don’t hand-edit tags. Create a py_torch_test macro that automatically adds "torch" to the tags list. Then tell Gazelle to map py_test to your macro for torch-dependent directories.

But there’s a subtlety — Gazelle will still try to generate its own py_test targets alongside your macro calls. So you also need # gazelle:exclude test_*.py in those BUILD files to prevent conflicts.

This is a recurring pattern with Gazelle: you can’t fight the code generator. You either make the generator do what you want, or you tell it to stay out of your way entirely.

Bump #4: The antlr4 sdist Catastrophe

omegaconf depends on antlr4-python3-runtime. That package has no wheel on PyPI — it’s sdist-only. Normally, aspect_rules_py can build sdists. But in our Docker CI environment, with the local_runtime_repo toolchain, the sdist build rule hits a null pointer:

'NoneType' value has no field or method 'to_list'

The py_toolchain.files attribute is None with local toolchains. The sdist build rule doesn’t handle that case. It’s not a misconfiguration — it’s a genuine gap in the rule implementation.

The fix is a two-part workaround:

Part 1: Tell Bazel to stop trying to build antlr4 from source. Use uv.override_requirement in MODULE.bazel to replace the pip package with an empty Bazel target:

uv.override_requirement(
    hub_name = "pypi",
    requirement = "antlr4-python3-runtime",
    target = "//bazel:empty_py_lib",
    venv_name = "default",
)

Part 2: Pre-install antlr4 (and omegaconf) in the Docker image so they’re available at runtime:

RUN pip install --no-cache-dir --break-system-packages \
    omegaconf antlr4-python3-runtime

The empty_py_lib target is literally an empty py_library. Bazel thinks the dependency is satisfied. The Docker image provides the actual package at runtime. It’s not elegant. It works.

Bump #5: The Isolation Wall

This is the big one. The fundamental tension between Bazel hermeticity and system-installed native packages.

py_test runs Python with the -I flag — isolated mode. System site-packages are invisible. This is great for hermeticity: your test can only see what Bazel explicitly provides. If you forgot to declare a dependency, the test fails. Clean.

But torch is system-installed in Docker (because building it from source in CI would take hours and require CUDA toolkits). With -I mode, Python can’t see it:

ModuleNotFoundError: No module named 'torch'

Your Bazel dependency graph says “I don’t manage torch.” Your Docker image says “torch is right here.” Python’s -I flag says “I can only see what Bazel gives me.” Stalemate.

The fix: py_venv_test — a different test rule from aspect_rules_py that creates a virtual environment for each test. Crucially, it has a knob:

include_system_site_packages = True

This strips the -I flag. System packages become visible. torch works.

I wrapped this into the py_torch_test macro:

def py_torch_test(name, srcs = [], deps = [], args = [], tags = [], **kwargs):
    if "torch" not in tags:
        tags = tags + ["torch"]
    if "@pypi//pytest" not in deps:
        deps = deps + ["@pypi//pytest"]

    src_args = [native.package_name() + "/" + src for src in srcs]

    py_venv_test(
        name = name,
        srcs = srcs + [Label("//bazel/rules:pytest_runner.py")],
        main = Label("//bazel/rules:pytest_runner.py"),
        imports = ["."],
        include_system_site_packages = True,
        include_user_site_packages = False,
        target_compatible_with = ["@platforms//os:linux"],
        tags = tags,
        deps = deps,
        args = src_args + args,
        **kwargs
    )

This does several things at once:

• Always adds the "torch" tag (so CI filtering works)
• Restricts to Linux (torch tests only run in Docker, which is Linux)
• Makes system site-packages visible
• Uses a custom pytest runner

The two-tier architecture crystallized here: hermetic for pure Python (py_test), system-visible for native deps (py_venv_test in Docker).

Bump #6: The Label Resolution Trap

This one was genuinely sneaky. The py_torch_test macro references a shared pytest_runner.py file that lives in bazel/rules/. In the macro, I initially wrote:

main = "pytest_runner.py",

The test in mai_config/tests/BUILD.bazel calls the macro. Bazel resolves that main path relative to the caller’s package — mai_config/tests/ — not the macro’s package. So it looks for mai_config/tests/pytest_runner.py. Which doesn’t exist.

The error is wonderfully unhelpful:

python3: can't find '__main__' module in '.../runfiles/_main'

Nothing about file resolution, nothing about the actual path it tried. Just “can’t find main.”

The fix: Use Label(), which resolves relative to the .bzl file’s own package:

main = Label("//bazel/rules:pytest_runner.py"),

This is the idiomatic Bazel pattern for macros referencing their own files. aspect_rules_py itself uses Label() internally for the same reason. I knew this pattern existed. I still wrote the wrong thing first. That’s how traps work — you know about them in the abstract and still step in them in practice.

Bump #7: The Growing Docker Dependency List

Once system site-packages are visible, every system-installed package your code touches needs to actually be in the Docker image. The initial image had torch and omegaconf. That wasn’t enough.

Tests started failing with import errors for:

• hydra-core — mai_config imports it at module level
• torchaudio — audio_segmentation tests need it
• datadog — mai_logger metrics tests

Each missing package was its own CI push because the error only surfaces when that specific test runs. You fix one, push, wait for CI, discover the next one.

The final Dockerfile additions:

RUN pip install --no-cache-dir --break-system-packages \
    omegaconf antlr4-python3-runtime hydra-core
RUN pip install --no-cache-dir --break-system-packages \
    torchaudio --index-url https://download.pytorch.org/whl/cpu
RUN pip install --no-cache-dir --break-system-packages \
    datadog

Note torchaudio needs the PyTorch CPU wheel index — the default PyPI version expects CUDA, and our CI Docker image uses CPU-only torch for most tests.

The lesson: when you open the system site-packages door, you’re responsible for everything behind it. There’s no partial visibility.

Bump #8: The Coverage Checker Doesn’t Know Your Rules

Our CI has a check_bazel_coverage.py script that ensures every .py file in the repo is tracked by at least one Bazel target. It queries Bazel for all targets matching specific rule kinds:

kind('py_library|py_test|py_binary', //...)

py_venv_test is a different rule kind. Our new test targets are invisible to the coverage checker. Files appear “untracked” and CI fails with a coverage gap error.

The fix — one line:

kind('py_library|py_test|py_binary|py_venv_test', //...)

This is the kind of bug that makes you feel dumb and smart at the same time. Dumb because it’s obvious in hindsight. Smart because you did find it by reading the script instead of flailing.

Bump #9: Data Files Don’t Travel Automatically

audio_segmentation has a norm_config.py that reads punctuations.lst — a data file sitting next to it in the source tree. The code uses __file__-relative path resolution:

data_path = Path(__file__).parent / "punctuations.lst"

In a normal Python environment, this works because the file is right there. In Bazel, test execution happens in a runfiles directory — a sandbox containing only the files Bazel knows about. Gazelle scans Python imports, not open() calls. It has no idea punctuations.lst exists.

FileNotFoundError: No such file or directory: '.../punctuations.lst'

The fix: Explicitly declare the data file in the BUILD target:

py_library(
    name = "audio_segmentation",
    srcs = [...],
    data = ["punctuations.lst"],
)

Now Bazel copies it into runfiles. __file__-relative resolution finds it. This is one of those Bazel things that’s “working as designed” but catches everyone at least once — you have to declare all your files, not just code.

Bump #10: Redis Wants a Server

rocket_lib has Redis integration tests. They assume a running Redis server. Docker CI doesn’t have one.

This is the simplest bump on the list: tag them integration and add -integration to the Docker CI filter:

py_torch_test(
    name = "test_redis_cache",
    srcs = ["test_redis_cache.py"],
    tags = ["integration"],
    deps = [...],
)

The tag ends up as ["torch", "integration"] (the macro adds torch automatically). The Docker CI filter --test_tag_filters=torch,-integration includes it by the torch tag, then excludes it by the integration tag. Net result: skipped. These tests will run in a future integration test job that has actual infrastructure.

The Push-by-Push Timeline

Here’s how the 9 CI pushes played out, compressed:

Nine pushes. The first four were “getting the basics right” — exclusion layers, Gazelle config, import formatting. Push 5 was the pivot to the two-tier architecture. Pushes 6-9 were the tail of cascading discoveries — each fix revealing the next problem.

The Architecture That Emerged

Non-Docker CI:                    Docker CI:
+--------------+                 +----------------------+
| py_test      |                 | py_venv_test         |
| (isolated)   |                 | (system site-pkgs)   |
|              |                 |                      |
| Bazel deps Y |                 | Bazel deps Y         |
| System pkgs N|                 | System pkgs Y        |
|              |                 | (torch, omegaconf,   |
| tag: -torch  |                 |  hydra, torchaudio,  |
+--------------+                 |  datadog)            |
                                 | tag: torch           |
                                 +----------------------+

Two test environments. Two rule types. One tag-based routing system. The key insight: don’t fight Bazel’s hermeticity, route around it. Pure Python tests stay hermetic. Torch tests get their own lane with system visibility.

What I’d Do Differently

1. Map the Docker dependency surface first. Instead of discovering missing Docker packages one CI push at a time, I should have traced every import reachable from the six packages and cross-referenced with @pypi// targets. Anything imported but not in @pypi// needs to be in Docker.

2. Test locally with --test_tag_filters early. The CI tag routing is central to the whole scheme. I should have verified it locally before burning CI cycles.

3. Read the coverage checker before pushing. The py_venv_test kind issue was entirely predictable if I’d looked at the script beforehand.

4. Keep a running checklist of “things Gazelle doesn’t auto-detect.” Data files, integration tags, macro-specific attributes — these all need manual BUILD file edits. Having a checklist would’ve batched several fixes into one push.

The Bigger Picture

Six packages migrated. The torch-dependent frontier of our monorepo is no longer a Bazel dead zone. The two-tier architecture — hermetic py_test for pure Python, system-visible py_venv_test for native deps — scales to any package that needs CUDA, audio processing, or other system libraries.

The 10 bumps weren’t random. They form a pattern:

• Bumps 1-3: Tooling conflicts (Gazelle + ruff + CI tags). When your build system, linter, and code generator all have opinions about the same lines of code, they will disagree.
• Bump 4: Supply chain edge case (sdist-only package + local toolchain). The long tail of Python packaging never ends.
• Bumps 5-6: The hermeticity boundary (isolation mode + macro resolution). The core architectural challenge.
• Bumps 7-9: The completeness problem (Docker deps + coverage + data files). Once you open one door, you have to furnish the whole room.
• Bump 10: Scope management (integration tests need infrastructure). Not every test belongs in every CI job.

Each bump was individually solvable. The hard part was that they’re sequential — you can’t see bump 7 until you’ve fixed bump 6. It’s a dependency chain of failures, which is ironic for a dependency management system.

But the packages are migrated. The tests are green. And the next six packages will be easier, because the py_torch_test macro, the Docker image, and the coverage checker now know about the two-tier world.

That’s how migration works. Not in one clean push, but in nine messy ones where each failure teaches you something about the gap between your mental model and reality.

The tools referenced in this post: aspect_rules_py, Gazelle for Python, uv. The Label() function documentation is at bazel.build/rules/lib/builtins/Label — read it before writing your first macro.