The Full Chain: How Python BUILD Files Get Generated in a Bazel Monorepo

You run one command:

bazel run //bazel:gazelle

And BUILD files appear. py_library targets, py_test targets, dependencies resolved, everything wired up. Magic.

Except it’s not magic. It’s six layers of software, three different GitHub organizations, a YAML manifest with 860 entries, and a custom macro that splits tests by platform. I traced every layer because I needed to customize one of them, and you can’t customize what you don’t understand.

This is the full chain, bottom to top. Every component, how it connects to the next one, and where to look when something breaks.

The stack at a glance

┌─────────────────────────────────────────────────┐
│  You run: bazel run //bazel:gazelle             │  ← the command
├─────────────────────────────────────────────────┤
│  Aspect Gazelle (pre-built binary)              │  ← BUILD file generator
│  ├── reads Python source files (imports)        │
│  ├── consults gazelle_python.yaml (manifest)    │
│  └── emits py_library / py_test / py_binary     │
├─────────────────────────────────────────────────┤
│  gazelle:map_kind directives                    │  ← rule remapping
│  ├── py_library → @aspect_rules_py//py:defs.bzl │
│  ├── py_binary  → @aspect_rules_py//py:defs.bzl │
│  └── py_test    → //bazel/rules:defs.bzl        │  ← custom macro!
├─────────────────────────────────────────────────┤
│  aspect_rules_py (rules_py) v1.8.4              │  ← Python execution rules
│  ├── py_library, py_binary, py_test             │
│  ├── py_venv_test (virtual env test runner)     │
│  └── uv extension (reads uv.lock directly)      │
├─────────────────────────────────────────────────┤
│  rules_python v1.7.0                            │  ← Python toolchain
│  ├── Hermetic Python 3.12.2 toolchain           │
│  └── local_runtime_repo (system Python fallback)│
├─────────────────────────────────────────────────┤
│  uv.lock + pyproject.toml                       │  ← dependency source of truth
└─────────────────────────────────────────────────┘

Six layers. Three upstreams (rules_python, aspect_rules_py, aspect-gazelle), three local (gazelle_python.yaml, custom macro, patches). Let’s walk each one.

Layer 1: rules_python — the Python toolchain

At the very bottom, someone needs to give Bazel a Python interpreter. That’s rules_python.

# MODULE.bazel
bazel_dep(name = "rules_python", version = "1.7.0")

python = use_extension("@rules_python//python/extensions:python.bzl", "python")
python.toolchain(
    python_version = "3.12.2",
)

This registers a hermetic Python 3.12.2 — Bazel downloads and manages this interpreter itself. No system Python involved. Same version on every machine, every CI runner, every developer laptop. That’s the point.

rules_python also provides the basic py_library, py_binary, and py_test rules. We don’t use those directly — Aspect’s enhanced versions replace them — but they’re the foundation everything builds on.

The system Python escape hatch

Hermetic is great until you need torch. PyTorch is 2GB, ships platform-specific CUDA binaries, and does not play well with Bazel’s download-and-sandbox model. Our solution: register a second Python toolchain that points at the system-installed Python inside Docker images where torch is pre-installed.

local_runtime_repo = use_repo_rule(
    "@rules_python//python/local_toolchains:repos.bzl",
    "local_runtime_repo",
)

local_runtime_repo(
    name = "system_python3",
    dev_dependency = True,
    interpreter_path = "/usr/local/bin/python3",
    on_failure = "skip",
)

register_toolchains(
    "@local_toolchains//:all",
    dev_dependency = True,
)

The on_failure = "skip" is important — on macOS where there’s no /usr/local/bin/python3 matching the Docker image, this toolchain silently doesn’t register. Bazel falls back to the hermetic one. In CI Docker containers where system Python has torch pre-installed, this toolchain takes priority.

Two toolchains. Hermetic for everything normal, system for torch. The custom py_test macro (layer 6) decides which one to use.

Wait — why can’t torch just use the hermetic toolchain?

Fair question. Every other package downloads a wheel and runs in the sandbox. Why is torch special?

Three problems collide:

1. The lockfile excludes torch on Linux entirely. pyproject.toml has "torch; sys_platform == 'darwin'" — torch is only resolved for macOS. On Linux, uv.lock literally has no torch wheel to fetch. The uv extension can’t download what doesn’t exist.

2. CUDA coupling. Docker CI images ship torch pre-compiled against specific CUDA drivers, cuDNN, and NCCL versions. The PyPI torch wheel is either CPU-only or CUDA-generic — it doesn’t match the Docker image’s GPU stack. Using the wrong torch against the wrong CUDA is a fast path to CUDA error: no kernel image is available for execution on the device.

3. Sandbox isolation. Even if you could get the right wheel, Bazel’s hermetic Python runs in an isolated sandbox. It can’t see system-installed CUDA libraries. The system torch is specifically built against the Docker image’s CUDA stack — you need the system Python to access it.

So the design is: on Linux CI, don’t fetch torch at all — use the pre-installed one via system Python. On macOS (developer laptops), the PyPI torch wheel works fine — no CUDA needed for local CPU testing.

But wait — rules_python vs rules_py?

You might notice we have both rules_python v1.7.0 and aspect_rules_py v1.8.4. If one “replaces” the other, why keep both?

They’re complementary layers, not alternatives. Think of it as:

rules_python = “here’s a Python interpreter and how to find packages.” aspect_rules_py = “here’s a better way to run Python code and resolve deps from uv.lock.”

If you removed rules_python, there’d be no Python toolchain, no system Python registration, and no gazelle Python support. Everything breaks. aspect_rules_py builds on top of rules_python — it doesn’t replace the foundation.

Layer 2: aspect_rules_py — enhanced Python rules

aspect_rules_py replaces the basic rules_python rules with better versions. The headline feature for us is py_venv_test — a test rule that creates a proper virtual environment, which lets us expose system site-packages (i.e., the system torch).

# MODULE.bazel
bazel_dep(name = "aspect_rules_py", version = "1.8.4")

single_version_override(
    module_name = "aspect_rules_py",
    patch_strip = 1,
    patches = [
        "//bazel/patches:aspect_rules_py_hashless_wheels.patch",
        "//bazel/patches:aspect_rules_py_sdist_build_helper.patch",
        "//bazel/patches:aspect_rules_py_sdist_env.patch",
    ],
)

Three patches on top of the upstream release. Each one fixes a real pain point:

These are the kind of patches you accumulate when running Bazel Python at scale against a real monorepo. Upstream works for the 90% case. The patches handle the other 10%.

Layer 3: the uv extension — dependency resolution from uv.lock

Here’s where Python packages become Bazel targets.

The monorepo uses uv for dependency management. All dependencies are declared in pyproject.toml and pinned in uv.lock. The aspect_rules_py uv extension reads uv.lock directly and creates a Bazel “hub” — a virtual repository where every pinned package is a target.

# MODULE.bazel
uv = use_extension("@aspect_rules_py//uv/unstable:extension.bzl", "uv")
uv.declare_hub(hub_name = "pypi")
uv.declare_venv(
    hub_name = "pypi",
    venv_name = "default",
)
uv.lockfile(
    src = "//:uv.lock",
    hub_name = "pypi",
    venv_name = "default",
)

After this, @pypi//requests, @pypi//torch, @pypi//numpy — all valid Bazel targets. The uv extension handles downloading wheels, extracting them, and making them available as py_library dependencies. No pip install. No requirements.txt parsing. The lockfile is the single source of truth.

Sdist annotations

Most packages ship pre-built wheels. Some don’t. When a package only ships source distributions (sdists), it needs build-time dependencies to compile. Those are declared in a TOML file whose format is defined by aspect_rules_py’s uv.unstable_annotate_requirements() (note the unstable_ prefix — this API isn’t finalized):

# bazel/pip_annotations.toml
[[package]]
name = "fasttext"
build-dependencies = [
    { name = "pybind11" },
    { name = "numpy" },
    { name = "wheel" },
]

Without this, Bazel tries to build fasttext without pybind11 in the sandbox and you get a cryptic C++ compilation error. With it, the build deps are fetched first and made available during the sdist build. This format is not from rules_python — it’s aspect_rules_py-specific.

Layer 4: Gazelle — the BUILD file generator

Now we’re at the layer most people interact with. Gazelle is a BUILD file generator — it walks your source tree, parses your code, figures out what depends on what, and writes BUILD files for you.

Our setup uses Aspect’s pre-built gazelle binary rather than compiling from Go source. This binary bundles the Python language plugin and runs significantly faster. The binary is managed by rules_multitool — a Bazel rule for managing pre-built tool binaries. It defines the lockfile format:

// bazel/tools/tools.lock.json (schema by rules_multitool)
{
  "$schema": "https://raw.githubusercontent.com/theoremlp/rules_multitool/main/lockfile.schema.json",
  "gazelle": {
    "binaries": [
      {
        "url": "https://github.com/aspect-build/aspect-gazelle/releases/download/2025.40.171+aa8a243c/gazelle-darwin_arm64_cgo",
        ...
      }
    ]
  }
}

The gazelle target itself:

gazelle(
    name = "gazelle",
    extra_args = GAZELLE_DIRS,
    gazelle = "@multitool//tools/gazelle",
)

GAZELLE_DIRS = [
    "apps/dataplatform/leaderboard/",
    "ci/",
    "lib/mai_nano/",
    # ... 24 directories total
]

GAZELLE_DIRS is the migration boundary. Gazelle only processes directories on this list. As packages get migrated to Bazel, they get added here. Currently 24 directories — about 40% of the monorepo. The rest still builds with the old system.

What gazelle does when it runs

For each Python file in GAZELLE_DIRS:

1. Parse imports — AST-level, not regex. It reads import foo and from foo.bar import baz and builds a list of dependencies.
2. Classify each import:
   • Standard library (os, sys, json) → ignored
   • First-party (another package in the monorepo) → //path/to:target
   • Third-party (PyPI package) → @pypi//package_name
3. Group by directory — all Python files in a directory become one py_library, test files become py_test targets
4. Write BUILD.bazel — or update the existing one, preserving manually-added directives

The third-party resolution is where the manifest comes in.

Layer 5: gazelle_python.yaml — the module mapping manifest

Python has a naming problem. The package you pip install is not always the thing you import. You install Pillow, you import PIL. You install PyYAML, you import yaml. You install scikit-learn, you import sklearn.

Gazelle sees import PIL in your source code and needs to figure out that it should add @pypi//pillow to your BUILD file. That mapping lives in gazelle_python.yaml:

# bazel/gazelle_python.yaml
manifest:
  modules_mapping:
    PIL: pillow
    cv2: opencv_python_headless
    yaml: PyYAML
    bs4: beautifulsoup4
    git: GitPython
    dotenv: python_dotenv
    jwt: PyJWT
    sklearn: scikit_learn
    google.protobuf: protobuf
    # ... 850+ more entries
  pip_repository:
    name: pypi
integrity: 6820e7f71a67a07e009fe1d0ca54115f062975596bbcb146920cc90fae83b33c

~860 entries mapping import names to package names. The format was defined by rules_python_gazelle_plugin — the Go extension that teaches gazelle to understand Python. Aspect’s gazelle binary is compatible with the same schema. The integrity hash at the bottom is a checksum — if the manifest is out of sync with the actual packages, gazelle warns you.

In theory, this manifest should be auto-generated from uv.lock — every wheel contains metadata listing what modules it provides. In practice, there’s an upstream bug that makes auto-generation unreliable, so we maintain it semi-manually. When you add a new PyPI dependency, you might need to add a line here if the import name doesn’t match the package name.

Layer 6: the custom py_test macro — platform-aware torch splitting

This is the local customization layer. Gazelle’s Python plugin always emits the canonical rule names — py_test, py_library, py_binary — regardless of what’s in your MODULE.bazel. It’s hardcoded. Even if you removed rules_python entirely, gazelle would still write py_test.

That’s where map_kind comes in. It’s a post-generation redirect — gazelle writes the standard name, and map_kind changes where the symbol loads from:

# Root BUILD.bazel
# gazelle:map_kind py_test py_test //bazel/rules:defs.bzl

Clean separation: gazelle decides what targets to generate, map_kind decides which implementation to use. You never need to fork gazelle to use custom rules.

Our macro wraps the upstream py_test and adds two things:

1. Automatic pytest injection:

def py_test(name, srcs = [], deps = [], data = [], args = [], tags = [], **kwargs):
    if _PYTEST_DEP not in deps:
        deps = deps + [_PYTEST_DEP]
    # ...

Every test gets @pypi//pytest as a dependency automatically. Developers don’t need to declare it.

2. Torch detection and platform splitting:

_TORCH_DEP = "@pypi//torch"

def py_test(name, srcs, deps, ...):
    if _TORCH_DEP in deps:
        _py_torch_test(...)
    else:
        _py_test(name, pytest_main=True, ...)

When gazelle sees import torch in a test file, it adds @pypi//torch to deps. The macro detects this and generates three targets instead of one:

def _py_torch_test(name, srcs, deps, ...):
    # Linux: use system Python (Docker has torch pre-installed)
    py_venv_test(
        name = name + "_linux",
        include_system_site_packages = True,
        target_compatible_with = ["@platforms//os:linux"],
        # @pypi//torch STRIPPED from deps — torch comes from system
    )

    # macOS: use hermetic Python + PyPI torch wheel
    _py_test(
        name = name + "_macos",
        target_compatible_with = ["@platforms//os:macos"],
        # @pypi//torch KEPT in deps — downloaded from PyPI
    )

    # Umbrella: platform-agnostic name that "does the right thing"
    native.test_suite(
        name = name,
        tests = [":" + name + "_linux", ":" + name + "_macos"],
        tags = ["torch"],
    )

The test_suite is the ergonomic glue. Without it, bazel test :test_foo would fail — only _linux and _macos exist. With it, you get a platform-agnostic name. Bazel runs the suite, silently skips the incompatible platform target, and the developer never thinks about it.

Here’s what actually happens on each platform:

Note: macOS does get a working torch — the PyPI CPU wheel. You can run torch tests locally. The only thing you can’t test on macOS is CUDA-specific behavior (GPU kernels, multi-GPU, etc.). For that you need the Docker image.

The developer writes import torch in their test. They don’t think about toolchains or platforms. The macro handles it. That’s the whole point.

Gazelle directives — the control knobs

Gazelle’s behavior is controlled by comment directives in BUILD.bazel files. The root BUILD file sets the global defaults:

# gazelle:python_generation_mode package
# gazelle:python_manifest_file_name bazel/gazelle_python.yaml
# gazelle:map_kind py_library py_library @aspect_rules_py//py:defs.bzl
# gazelle:map_kind py_binary py_binary @aspect_rules_py//py:defs.bzl
# gazelle:map_kind py_test py_test //bazel/rules:defs.bzl
# gazelle:resolve py torch @pypi//torch
# gazelle:exclude .venv
# gazelle:exclude mai_config/**

Let’s break these down:

Per-package directives let individual directories customize behavior:

# In a package's BUILD.bazel:
# gazelle:python_root          ← this directory is an importable root
# gazelle:python_default_visibility //visibility:public

And import overrides handle cases where gazelle guesses wrong:

# gazelle:resolve py caas_client.internal //caas_client/src/caas_client

This tells gazelle: when you see import caas_client.internal, don’t try to auto-resolve it — the target is //caas_client/src/caas_client.

The dependency chain — how it all flows

Here’s the same information as a data flow diagram. Follow the arrows to see how a new PyPI package goes from pyproject.toml to a resolvable Bazel target:

pyproject.toml  ──uv lock──▶  uv.lock  ──uv extension──▶  @pypi hub
    (deps)                     (pinned)    (aspect_rules_py)  (Bazel targets)
                                              │
                                              ▼
                               gazelle_python.yaml ◀── (semi-manual currently)
                               (import → package map)
                                              │
                                              ▼
  Python source  ──gazelle──▶  BUILD.bazel files
  (*.py files)     (parses      (py_library, py_test)
                    imports)          │
                                     │ loads
                                     ▼
                            ┌─────────────────────────┐
                            │  @aspect_rules_py//py    │  (py_library, py_binary)
                            │  //bazel/rules:defs.bzl  │  (py_test custom macro)
                            └─────────────────────────┘
                                     │
                                     │ uses
                                     ▼
                            ┌─────────────────────────┐
                            │  rules_python toolchain  │
                            │  ├── hermetic 3.12.2     │
                            │  └── system Python       │
                            │      (torch in Docker)   │
                            └─────────────────────────┘

The end-to-end workflow

When you’re migrating a new package to Bazel, here’s the actual sequence of operations:

Step 0: Prerequisites. uv.lock has all the package’s dependencies. gazelle_python.yaml has mappings for any imports where the import name differs from the package name. The directory is in GAZELLE_DIRS.

Step 1: Mark the import root. Add # gazelle:python_root to the package’s BUILD.bazel. This tells gazelle that import mypackage should resolve to this directory.

Step 2: Run gazelle.

bazel run //bazel:gazelle

Behind the scenes:

1. Bazel downloads the Aspect Gazelle binary (cached after first run)
2. Gazelle walks the directories in GAZELLE_DIRS
3. For each .py file, it parses imports at the AST level
4. Each import gets classified: stdlib (ignored), first-party (//path:target), or third-party (@pypi//pkg via the manifest)
5. Imports are grouped per directory into py_library and py_test targets
6. map_kind directives remap rule implementations to Aspect’s (and our custom macro)
7. BUILD.bazel files are written or updated

Step 3: Verify.

bazel build //path/to/package/...
bazel test //path/to/package/...

Component reference

Everything in one table:

When things break — a debugging guide

The chain has a lot of links. When something fails, the question is always: which link broke?

“Gazelle can’t resolve import X”

The manifest is missing a mapping. Check bazel/gazelle_python.yaml — does the import name have an entry? If not, add X: package_name. If the import name matches the package name (like import requests), check that the package is in uv.lock.

“I added a new package but @pypi//foo doesn’t exist”

The uv extension hasn’t picked it up. Run uv lock to update the lockfile, then bazel run //bazel:gazelle again. If the package only ships sdists, you may also need a pip_annotations.toml entry with its build dependencies.

“Test needs torch but Bazel can’t find it”

Just import torch in the test file and run gazelle. Gazelle adds @pypi//torch to deps. The custom macro detects this and auto-splits into Linux (system torch) and macOS (PyPI torch) targets. On macOS, the CPU-only PyPI wheel is fetched automatically — local dev works. For CUDA-specific testing, you’ll need the Docker image.

“My package should be migrated but gazelle ignores it”

Check GAZELLE_DIRS in the root BUILD file — is the directory listed? Also check for # gazelle:exclude directives that might be blocking it. And make sure there’s a # gazelle:python_root directive in the package’s BUILD.bazel.

“Gazelle generated wrong dependencies”

Check for # gazelle:resolve overrides in BUILD.bazel files. Gazelle infers dependencies from imports, but some patterns confuse it — dynamic imports, conditional imports, imports inside try/except blocks. Use # gazelle:resolve py module_name //target to manually override.

The aha moment

I started this investigation because I needed to customize how torch tests run. I expected to find one or two moving parts. Instead I found six layers, three upstream projects, three local customization points, and 860 lines of import mappings.

But here’s the thing — once you see the full chain, each individual layer is straightforward. rules_python gives you a Python. aspect_rules_py gives you better rules and reads your lockfile. Gazelle parses your imports and writes BUILD files. The manifest maps import names to package names. The custom macro handles the torch edge case.

No single layer is complicated. The complexity is in the composition — knowing that all these layers exist and how they connect. Once you have that map, customizing any piece of it becomes a matter of knowing which file to edit.

That’s what this post is. A map.