GPU Tests in Bazel: The Resource Problem Nobody Warned You About

2026/02/24

BUILDbazeltorch

TL;DR: I spent several posts framing “torch in Bazel” as a dependency resolution problem. It is. But a team meeting surfaced the other hard problem: resource management. Even if you solve dependency resolution perfectly, Bazel has no concept of “this test needs 2 GPUs and 40GB of VRAM.” Neither does its remote execution backend. The team’s pragmatic answer: Bazel decides what to test, a custom planner decides how to run it. GPU test orchestration via Bazel is probably not worth it.

This post is a companion to What Does Success Look Like? — specifically expanding on why Level 5 (GPU tests via Bazel) was marked “Dream (and probably not worth it).” It also connects to Test Classification, which is what makes the pragmatic approach possible.

The Conversation That Changed the Frame

In a team meeting, someone asked the question directly: “Do we really want to run GPU tests using Bazel?”

The honest answer from our tech lead: “That’s a question I don’t want to make a decision on yet. I do not know how to run GPU tests efficiently on Bazel.”

Not “no.” Not “yes, here’s the plan.” Just: I don’t know how, and I’m not going to pretend I do.

His practical position: “I’m happy to say, let’s use Bazel to decide which tests to run, and then use our custom planner to figure out how to run them.”

That sentence — Bazel decides what, planner decides how — reframed everything for me. I’d been thinking about torch-in-Bazel as one problem. It’s actually two, and the second one is harder than the first.

The Iceberg

Here’s the mental model:

 ┌──────────────────────────────────────┐
 │                                      │  ← What I've been writing about
 │      Dependency Resolution           │     (torch resolves on Linux,
 │      (the visible problem)           │      lockfiles, overrides, etc.)
 │                                      │
 ├──────────────────────────────────────┤  ← waterline
 │                                      │
 │      Resource Management             │  ← What this post is about
 │      (the underwater problem)        │     (GPUs as a schedulable resource,
 │                                      │      memory, CUDA versions, topology)
 │                                      │
 │                                      │
 └──────────────────────────────────────┘

The dependency resolution problem is: how does @pypi//torch resolve to a working binary on Linux? Lockfile strategies, override mechanisms, hermetic vs system Python. I’ve written thousands of words about this. It’s a real problem, and we’re making progress on it.

The resource management problem is: even if torch resolves, builds, and imports perfectly — how does Bazel know that test_multi_gpu_training needs 4 GPUs with 80GB of VRAM each, and test_tensor_shape needs zero GPUs and 200MB of RAM?

Bazel knows about CPU cores and RAM. It has --local_resources=cpu=8,memory=16384. There is no --local_resources=gpu=4,vram=320000. There’s no built-in concept of “GPU” as a schedulable resource.

The BuildBarn Story

A teammate who dealt with this at a previous company shared his experience with BuildBarn — an open-source implementation of Bazel’s Remote Execution API. Think of it as “Bazel’s cloud backend”: you submit build and test actions, BuildBarn dispatches them to workers with the right capabilities.

For GPU tests, the idea sounds clean: Bazel decides what to test, BuildBarn dispatches GPU tests to workers that have GPUs. In practice:

That last point is the killer. BuildBarn workers are statically configured. You define a worker with certain capabilities (OS, CPU architecture, installed tools), and Bazel matches actions to workers based on those capabilities. But GPUs aren’t like CPUs. You can’t say “this worker has 8 GPU-cores” and let Bazel carve them up the way it carves up CPU time.

A GPU is more like a laboratory instrument than a CPU core. Each one has:

Bazel’s resource model was designed for fungible resources — CPU and RAM. A CPU core is a CPU core. A gigabyte of RAM is a gigabyte of RAM. GPUs are non-fungible in ways that matter for test correctness.

What We Actually Do Today

Our current GPU test setup is blunt but functional:

GPU request What you get
1 GPU 1 GPU on a shared node (one test at a time per GPU)
> 1 GPU 8 GPUs (the whole node)

One test per GPU at any given time. No simultaneous GPU tests sharing a GPU. If a test says it needs more than one GPU, we hand it an entire 8-GPU node rather than try to carve out exactly 2 or 4.

Is this wasteful? Yes. A test that needs 2 GPUs gets 8. That’s 6 idle GPUs. But it’s safe. No OOM kills from memory contention. No NCCL failures from unexpected GPU topology. No debugging sessions where the test passed locally but fails in CI because another test was using 30GB of the same GPU’s memory.

The alternative — packing multiple GPU tests onto one node — requires knowing exactly how much GPU memory each test uses. Not “approximately.” Exactly. Because if two tests that each need 35GB of a 40GB GPU run simultaneously, they both crash. Not “one succeeds and one fails.” Both crash. GPU memory isn’t like system RAM — there’s no swap file, no graceful degradation. You OOM and you die.

Why This Is Fundamentally Hard

Let me spell out the specific questions that “GPU tests in Bazel” requires you to answer:

1. How does Bazel know a test needs GPUs at all?

There’s no standard Bazel attribute for “this test requires GPU hardware.” You’d need either:

2. How does Bazel know how many GPUs?

A test that needs 1 GPU and a test that needs 8 GPUs have completely different scheduling requirements. With CPU/RAM, Bazel handles this through --local_resources. There’s no equivalent for GPUs. You’d need to encode GPU count in the test target somehow, and your scheduler would need to respect it.

3. How do you handle GPU memory?

Two tests that each need 1 GPU aren’t interchangeable if one needs 8GB of VRAM and the other needs 40GB. An A100 has 80GB — you could fit two 40GB tests, or ten 8GB tests. But Bazel doesn’t know about VRAM. Neither does BuildBarn. You’d need to build a GPU memory-aware scheduler from scratch.

4. How do you prevent memory contention?

Even if you solve scheduling perfectly, GPU memory fragmentation is real. A test that allocates and frees tensors in a specific pattern might leave the GPU in a state where 40GB is “free” but no single contiguous block larger than 20GB exists. The next test tries to allocate 30GB, gets an OOM, and everyone stares at the CI dashboard wondering why a test that needs 30GB failed on a GPU with 40GB free.

5. How do you handle CUDA version requirements?

Some tests need CUDA 12.4. Others work with CUDA 11.8. Your remote execution workers need to advertise which CUDA version they have, and Bazel needs to match tests to compatible workers. This is possible with execution platforms and constraints, but it’s another dimension of complexity on top of everything else.

6. What about multi-GPU topology?

Tests that use NCCL for multi-GPU communication care about how GPUs are connected — NVLink is 10x faster than PCIe for GPU-to-GPU transfers. A distributed training test that passes on NVLink-connected GPUs might timeout on PCIe-connected ones. Now you’re encoding network topology into Bazel scheduling constraints.

Each of these is solvable in isolation. Together, they form a scheduling problem that’s genuinely hard — harder than anything Bazel was designed to handle.

The Pragmatic Architecture

The team landed on a layered approach. Not because it’s elegant, but because it acknowledges what each tool is good at:

Layer 1: Bazel Owns the Graph (Now)

Bazel knows the dependency graph. It knows that test_model_forward.py depends on model_arch which depends on torch. It knows that test_config_parsing.py doesn’t touch torch at all. This information is valuable even if Bazel never runs a single GPU test.

“We want Bazel to know about GPU tests. We just don’t want Bazel to run GPU tests.”

That distinction — knowing vs running — is the whole insight. The dependency graph is Bazel’s superpower. GPU scheduling is not.

Layer 2: Classification Separates CPU from GPU (Soon)

This is the test classification work. If tests that import torch but don’t actually need a GPU are properly classified, they can run on CPU runners via Bazel. The ~40% of tests that only need CPU torch stop waiting for GPU runners.

Classification makes the pragmatic approach work because it minimizes what the GPU scheduler has to handle. Instead of all torch tests, it only handles tests that genuinely need GPU hardware.

Layer 3: A Custom Planner Handles GPU Scheduling (Soon-ish)

A purpose-built test planner that understands GPU resources. Not Bazel, not BuildBarn — a planner designed from the ground up for the specific constraints of GPU test scheduling.

It takes Bazel’s output (“these tests need to run”) and figures out the how: which GPU nodes, how many GPUs per test, memory requirements, CUDA version compatibility. It speaks the language of GPU infrastructure because that’s all it does.

The tech lead’s framing: “I have enough of a plan for test execution to take us through mid-cycle. I want to build what I’ve already designed, get that rolling. Then hook it up to Bazel. Then we assess whether we need fractional GPU nodes.”

Notice the ordering: build the planner first, hook it up to Bazel second. The planner doesn’t depend on Bazel. Bazel is an input source, not a requirement.

Layer 4: Maybe Fractional GPU Nodes (Later)

Once the planner exists, the question becomes: do we need to pack multiple tests onto one GPU node? If GPU runner queue times are acceptable with one-test-per-GPU, the answer is no. If queue times are painful, the planner could learn about GPU memory requirements and pack compatible tests together.

But this is optimization. You don’t optimize until you have data showing it’s needed.

Layer 5: Bazel Directly Orchestrating GPU Tests (Probably Never)

This was Level 5 in the success post. After this meeting, I’m more convinced than ever that it’s the wrong goal. Not because it’s technically impossible — you could build custom Bazel rules, extend BuildBarn with GPU-awareness, encode VRAM and CUDA versions as platform constraints. You could do all of that.

But you’d be building a GPU-aware scheduler inside a build system. And there are already people whose full-time job is building GPU-aware schedulers — they’re called Kubernetes, Slurm, and your cloud provider’s batch compute service. Rebuilding that capability inside Bazel because “we want one tool” is a trap.

BuildBarn: The Remote Execution Angle

It’s worth understanding why BuildBarn specifically doesn’t solve this, because it illustrates a broader point about remote execution backends.

BuildBarn implements the Bazel Remote Execution API. You can specify execution requirements for actions — things like “run this on Linux x86_64” or “run this in a container with Python 3.12.” You can even specify containers for BuildBarn workers, so in theory you could say “run this test inside a PyTorch container.”

But:

None of these are unsolvable. They’re all “another thing you have to build.” And at some point, the pile of “another thing” adds up to “you’re building a custom GPU orchestration platform that happens to use Bazel as an API.”

The Lesson

The torch-in-Bazel journey has two peaks:

Peak 1: Dependency resolution. How does @pypi//torch become a usable dependency on all platforms? This is a build system problem, and Bazel has the right primitives (lockfiles, overrides, platform constraints) to solve it. We’re climbing this one now.

Peak 2: Resource management. How do you schedule GPU tests on GPU hardware with GPU-specific constraints? This is an infrastructure problem, and Bazel’s primitives (CPU/RAM resources, static worker capabilities) are the wrong shape for it.

The pragmatic move is to recognize where each tool’s jurisdiction ends. Bazel is excellent at dependency graphs, incremental builds, test selection, and caching. It is not a GPU scheduler, and making it one would require building the GPU scheduling problem from scratch inside a build system that was designed for source code.

The layered architecture — Bazel for the graph, classification for the split, a custom planner for GPU scheduling — isn’t elegant. It’s three tools instead of one. But each tool does the thing it’s good at, and the interfaces between them are clean: Bazel outputs a list of tests to run and their types. The planner takes GPU tests and schedules them. Classification ensures only genuinely-GPU tests reach the planner.

Sometimes the right architecture is the one that acknowledges no single tool should do everything.

Where This Fits

The series arc, updated:

The dependency resolution posts were about the tip of the iceberg. This one is about what’s underneath. And the answer to “what’s underneath” turned out to be: a scheduling problem that Bazel was never designed to solve, and probably shouldn’t try to.

Next: actually building things instead of writing about why they’re hard.

Previous in series: Test Classification: The Real Torch CI Problem | Torch in Bazel: What Does Success Look Like?