The Test Hierarchy Map: pytest, Bazel, and Everything in Between

When you’ve been writing pytest for years, “test” means one thing. Then you start working with Bazel and realize “test” means something completely different — not wrong, just different. The word is the same. The granularity, the caching semantics, the selection mechanisms, the dependency tracking — all different.

This is the conceptual map I wish someone had drawn for me when I started working on Bazel migration. Not “how to write a BUILD file” — there are docs for that. More like: when someone says “run the tests,” what unit of work are they actually talking about? And how does that unit change depending on whether you’re thinking in pytest, Bazel, or your CI test selector?

The pytest hierarchy

Start with what we know. From smallest to largest:

Test function

def test_attention_forward():
    output = attention(query, key, value)
    assert output.shape == (batch_size, seq_len, d_model)

The atomic unit. pytest discovers these by naming convention — anything starting with test_. One assertion scope. This is the finest thing pytest can select, report on, or skip.

Parameterized test case

@pytest.mark.parametrize("batch_size", [1, 4, 16, 64])
@pytest.mark.parametrize("dtype", [torch.float32, torch.float16])
def test_attention_forward(batch_size, dtype):
    ...

One function, eight test cases. pytest reports each param combo separately: test_attention_forward[1-float32], test_attention_forward[64-float16], etc. This is the finest granularity pytest reports. You can select individual parameterizations with -k.

Test class

class TestAttentionLayer:
    def test_forward(self): ...
    def test_backward(self): ...
    def test_mask_handling(self): ...

Groups related test functions. Optional in pytest — unlike unittest, you don’t need a class. Shares setup/teardown via setup_method / teardown_method. Mostly an organizational choice.

Test file / module

test_attention.py — the collection unit. pytest collects by file. This is where conftest.py starts mattering: fixtures in a conftest.py are scoped to the directory tree it lives in. All tests below it in the filesystem get access. No declaration required.

Test suite / package

A directory of test files. In our monorepo, usually one per package: mai_layers/tests/, rocket/src/rocket/tests/. This is the level where our CI test selector find_changed operates — it says “run all tests in mai_layers,” not “run test_attention.py.”

The Bazel hierarchy

Different world, different units.

Test target

# BUILD file
py_test(
    name = "test_attention",
    srcs = ["test_attention.py"],
    deps = [
        "//mai_layers/src/mai_layers:attention",
        "@pypi//torch",
    ],
)

The atomic unit Bazel sees. One entry in a BUILD file. This is where Bazel caches, schedules, retries, and reports. A test target typically wraps one test file, but it could wrap many. Bazel doesn’t know or care what’s inside the file — it sees the target as a black box that exits 0 or 1.

This is the key difference. pytest’s atom is a function. Bazel’s atom is a target. A target usually contains many functions.

Test rule

py_test, py_venv_test, py_torch_test — the macro or rule that defines how a test runs. Do you need a virtual environment? System site-packages? A sandbox with torch available? The rule handles that.

# Our custom macro for torch tests
py_torch_test(
    name = "test_cuda_attention",
    srcs = ["test_cuda_attention.py"],
    deps = ["//mai_layers/src/mai_layers:attention"],
    tags = ["torch"],
    target_compatible_with = ["@platforms//os:linux"],
)

Rules aren’t a level in the hierarchy exactly — they’re more like the type system for test targets. Different rules impose different execution environments.

Test tags

py_test(
    name = "test_attention",
    tags = ["torch", "gpu", "slow"],
    ...
)

Metadata for filtering. bazel test //... --test_tag_filters=-slow runs everything except slow tests. Tags live on targets, not on individual test functions. You can’t tag test_forward as slow and test_backward as fast within the same target. The whole target is slow, or it isn’t.

Compare this to pytest marks, which live on functions:

@pytest.mark.slow
def test_forward_large_batch(): ...

def test_forward_small_batch(): ...  # not marked, runs by default

Same concept. Different scope. That difference matters when you have a target with 50 test functions and only 3 are slow.

Test suite

test_suite(
    name = "all_tests",
    tests = [":test_attention", ":test_embeddings", ":test_loss"],
)

Bazel’s concept of grouping targets. Rarely used directly — wildcards serve the same purpose: bazel test //mai_layers:all or bazel test //mai_layers/....

Package

A directory with a BUILD file. bazel test //lib/fortknox/... runs all test targets in that package and its subpackages. This is roughly equivalent to the pytest test suite level.

Workspace

bazel test //... — everything. The nuclear option. Also the thing that “just works” if your dep graph is complete, because Bazel only runs what actually changed.

The mapping problem

Here’s where it gets interesting. These hierarchies don’t map neatly onto each other. They’re not just different names for the same things.

The critical insight: Bazel’s finest granularity (test target ~ test file) is coarser than pytest’s finest granularity (test function), but finer than find_changed’s granularity (test package).

pytest function  <  Bazel target  <  find_changed package  <  "run everything"
   (finest)          (middle)          (coarsest)               (nuclear)

This is why the dep graph unlock matters. We go from package-level selection to target-level selection. Not function-level — nobody has a dep graph at that granularity — but target-level is the sweet spot.

A brief detour through Go

Go’s hierarchy is worth mentioning because it sits between Python and Bazel in interesting ways:

• func TestFoo(t *testing.T) — test function, discovered by naming convention (like pytest)
• t.Run("subtest", ...) — subtests, like pytest.mark.parametrize
• _test.go files live in the same package as production code (unlike Python’s separate test directories)
• go test ./... — run all, like bazel test //...

The key difference: Go compiles one test binary per package, and that binary runs all the test functions. Bazel works the same way — one target, one execution, many functions inside. pytest is the odd one out here, with its function-level selection and reporting. Python’s dynamic nature makes fine-grained selection possible. Go and Bazel’s compilation model makes it impractical.

What each level gives you

This is the table I keep coming back to. Different levels of the hierarchy give you fundamentally different capabilities:

Two things jump out:

Caching. pytest doesn’t cache. It runs everything you tell it to, every time. Bazel caches at the target level — if the target’s inputs (source files, deps, environment) haven’t changed, it doesn’t run. Cache hit. Done. This is the single biggest difference in practice. Going from “re-run everything” to “re-run nothing that hasn’t changed” is not an incremental improvement. It’s a phase transition.

Dependency tracking. pytest has no dep tracking. find_changed tracks at the package level via uv.lock. Bazel tracks at the target level via BUILD file deps. Finer dep tracking means fewer tests to run — you’re not dragging in the entire package because one file in it changed.

The conftest problem

This deserves its own section because it’s one of the weirdest mapping problems between the two worlds.

In pytest, conftest.py is magic scope. You drop a conftest.py in a directory, and every test below it in the filesystem tree gets access to its fixtures. No imports. No declarations. It just works.

mai_layers/
  tests/
    conftest.py          # fixtures available to ALL tests below
    test_attention.py    # uses fixtures, never imports conftest
    test_embeddings.py   # same — implicit access
    subdirectory/
      conftest.py        # additional fixtures, stacks with parent
      test_special.py    # sees both conftest.py files

This is convenient and also deeply implicit. There is no declaration anywhere that test_attention.py depends on conftest.py. pytest’s test collection walk handles it at runtime.

In Bazel, this becomes an explicit dependency:

# BUILD file
py_library(
    name = "test_fixtures",
    srcs = ["conftest.py"],
    deps = [...],
)

py_test(
    name = "test_attention",
    srcs = ["test_attention.py"],
    deps = [
        ":test_fixtures",     # explicit!
        "//mai_layers/src/mai_layers:attention",
    ],
)

Every test target that needs conftest fixtures must declare a dep on the library containing them. Forget the dep, and the test fails in the sandbox — conftest isn’t there, fixtures aren’t registered, tests error out.

This is Bazel doing its thing: making implicit dependencies explicit. It’s more work upfront. It’s also why Bazel can answer “what depends on this conftest?” instantly, while find_changed has to walk the directory tree.

Our find_changed tool handles this by walking upward from each test file, checking if any conftest.py in the chain changed. It works, but it’s a heuristic bolted onto a system that doesn’t natively understand the relationship. Bazel makes the relationship a first-class edge in the dep graph.

The parametrize multiplication problem

Here’s a granularity limitation that bites in practice.

@pytest.mark.parametrize("model_size", ["small", "medium", "large"])
@pytest.mark.parametrize("precision", ["fp32", "fp16", "bf16"])
@pytest.mark.parametrize("batch_size", [1, 4, 16, 64])
def test_model_forward(model_size, precision, batch_size):
    ...

That’s 3 * 3 * 4 = 36 test cases from one function. pytest sees 36 individually selectable, individually reportable test cases. Bazel sees one target.

If 35 pass and 1 fails, pytest can rerun just the failing parametrization. Bazel reruns the entire target — all 36. If each case takes 10 seconds, that’s 360 seconds of retry for a 10-second failure.

This isn’t a theoretical concern. In our monorepo, we have test files with hundreds of parametrized cases. The --flaky_test_attempts flag retries the whole target, not the failing case. For heavily-parametrized targets, the retry cost is real.

Workarounds exist but aren’t great:

• Split into multiple targets: One py_test per param group. Defeats the purpose of parametrize and inflates your BUILD file.
• pytest-rerunfailures inside Bazel: The target reruns failures internally before reporting to Bazel. Works, but now you have two retry systems.
• Accept it: For most targets, the total runtime is short enough that retrying all is fine. Reserve optimization for the genuinely expensive ones.

The honest answer is that this is a fundamental mismatch. pytest’s function-level granularity and Bazel’s target-level granularity see the world differently, and parametrize is where the difference hurts most.

What this means for a monorepo

The practical punchline. When someone changes a file in mai_layers, what happens at each level?

Package level (find_changed): “Run all tests in mai_layers.” That’s maybe 200 test cases across 15 files. Plus all tests in every downstream package. Could be thousands.

Target level (Bazel): “Run //mai_layers:test_attention and //mai_layers:test_embeddings.” That’s 2 files, maybe 40 test cases. Only the targets whose transitive deps include the changed file. Plus only the specific downstream targets that are affected, not entire downstream packages.

Function level (pytest with -k): “Run test_attention::test_forward_pass.” One test, one function. But nobody has a dep graph at function granularity. You’d need to track which functions call which functions, and Python’s dynamic dispatch makes that somewhere between hard and impossible.

Target-level is the sweet spot: fine enough to matter (3-10x fewer tests than package-level), coarse enough to maintain (one target per file, not one target per function). This is the granularity Bazel gives you for free once the dep graph is complete.

The selection spectrum

I keep thinking about this as a spectrum of precision vs. maintenance cost:

run everything → package selection → target selection → function selection
                 (find_changed)       (Bazel)            (pytest -k)

Less precise                                                More precise
Less maintenance                                           More maintenance

find_changed is low maintenance (package-level reasoning, one tool) but low precision (whole packages). Bazel is moderate maintenance (BUILD files per target, but gazelle auto-generates most of it) and high precision (individual targets). Function-level selection is theoretically possible but nobody maintains a dep graph at that granularity — the maintenance cost would be absurd.

The Bazel migration moves us one step to the right on this spectrum. Not all the way to function-level — that’s not practical. But from package-level to target-level. And that one step is where the 3-10x improvement lives.

This is a companion to What a Full Dep Graph Actually Unlocks, which covers the concrete CI wins of target-level selection. Previous related posts: Graduating Test Selection from find_changed to Bazel, Four Ways Bazel Breaks Your Python Tests.