Someone on the team opened a PR last week and pinged the CI support channel: “Can someone help me fix these Bazel issues?” The Bazel job was showing a big red ❌ on their PR. They spent 20 minutes trying to figure out what they broke.
They broke nothing. Bazel CI isn’t a required check. It’s advisory. But GitHub doesn’t have a way to show “this failed, and that’s fine” — it’s either green or red. So the PR page showed red, and a reasonable engineer did what reasonable engineers do: they tried to fix it.
This is the CI cried wolf problem. And the standard solutions are both bad.
The two bad options
Option A: Make it required. Your new CI gate blocks every PR. But the gate isn’t ready — coverage is incomplete, there are known failures, the infrastructure is still being hardened. You’ve just made every PR author responsible for fixing things they didn’t break. People start hating Bazel specifically rather than the growing pains of the migration. You’ve poisoned the well.
Option B: Leave it non-required. The check runs, fails, shows red. But since it’s not required, people learn to ignore it. Then when it does catch a real issue, nobody’s looking. You have a fire alarm that goes off every day at lunch — eventually people stop evacuating.
We tried Option B first. The Slack thread was the predictable result. Our tech lead pointed out that “required” is indicated by a small label on the right side of the checks panel. Nobody reads labels. People read ❌.
The soft-fail pattern
The fix we landed in PR #24181: every Bazel CI step runs with continue-on-error: true. The job always reports green ✅. Real failures get recorded as ::warning:: annotations and written to $GITHUB_STEP_SUMMARY.
- name: Bazel test
id: test
continue-on-error: true
run: |
bazel test //... 2>&1 | tee /tmp/bazel-test.log
- name: Report Bazel CI results
if: always()
run: |
# Check each step's outcome
if [[ "${{ steps.test.outcome }}" == "failure" ]]; then
echo "::warning::Bazel step 'test' failed (non-blocking)"
fi
# ... repeat for each step, write summary table
The job always shows green. The GitHub Step Summary contains the real results. And the ::warning:: annotations are the machine-readable signal.
Trade-off is obvious: you’ve gone from “red but ignorable” to “green but maybe lying.” We traded a false alarm for a hidden signal. That sounds worse when I put it that way, but in practice it solved the actual problem — people stopped pinging CI support about Bazel failures they didn’t cause.
The three-tier architecture
Quick context on what’s actually running. Our Bazel CI has three jobs:
jobs:
bazel: # CPU — fetch, lint, gazelle, build, coverage, test
bazel-torch-docker: # Torch CPU tests in lightweight Docker
bazel-gpu-docker: # GPU tests in pretrain_base image (condor2 runner)
The bazel and bazel-torch-docker jobs use the soft-fail pattern on every step. The GPU job is a different beast — it probes ACR for a pre-built image and skips entirely if it doesn’t exist:
TAG="main-${{ hashFiles('docker/Dockerfile', 'docker/pretraining/Dockerfile', 'docker-bake.hcl') }}"
docker manifest inspect "inf5acr.azurecr.io/pretrain_base:$TAG"
The image is built by a separate workflow on a 2-hour timer. The tag is a hash of the Dockerfiles, so if someone changes a Dockerfile, there’s a gap — up to 2 hours — where GPU tests silently skip because the new image hasn’t been built yet.
That’s a known limitation. More on those later.
The green lie
Here’s where it gets interesting. I was working on PR #25162 to re-enable Bazel GPU tests. Checked the CI status. All green. Every job passing. Great, right?
Not great.
I dug into the annotations via the GitHub API and found:
| Step | Real Status |
|---|---|
| gazelle | ❌ BUILD files out of date |
| build | ❌ failed (cascade from gazelle) |
| coverage | ❌ failed (cascade) |
| test | ❌ failed (cascade) |
| torch-test | ❌ separate failure |
| GPU probe | ⚠️ image not found (expected) |
Five out of eight steps actually failed. The GitHub UI showed all green. Zero indication that anything was wrong unless you knew to go looking.
This is the fundamental limitation of the soft-fail pattern: continue-on-error: true doesn’t just hide the failure from the PR author — it hides it from you, the person who needs to know.
How to actually check
The GitHub API tells the truth, even when the UI doesn’t. The trick is annotations.
When a step uses continue-on-error: true and fails, the API returns conclusion: "success" for the step — because from GitHub’s perspective, the step “succeeded” (it just happened to have a non-zero exit code that was intentionally swallowed). But the ::warning:: annotations from our report step are accessible:
gh api "repos/{owner}/{repo}/check-runs/{job_id}/annotations"
Returns something like:
[
{
"annotation_level": "warning",
"message": "Bazel step 'test' failed (non-blocking)",
"title": "Bazel CI: test FAILED"
}
]
That’s the real signal. The annotations contain messages like "Bazel step 'test' failed (non-blocking)" — structured enough to parse, human-readable enough to understand.
We built a CLI for it
Parsing annotations manually gets old fast, so we built a Claude Code slash command — /zpr-checkci-bazel — that automates the whole thing:
- Find the Bazel workflow run for a given PR
- Fetch job-level data from the GitHub API
- Parse annotations to get the real pass/fail per step
- Print a clear table
The output looks like:
PR #25162 — Bazel CI Real Status
─────────────────────────────────
✅ fetch passed
❌ gazelle BUILD files out of date
❌ build failed (cascade from gazelle)
❌ coverage failed (cascade)
❌ test failed (cascade)
❌ torch-test failed
⚠️ gpu-docker image not found (expected)
5/8 steps failed. Root cause: gazelle (BUILD files out of date)
Five steps failed, but only one root cause. Which brings up another problem.
Cascade failures are noisy
When gazelle fails — meaning your BUILD files are out of date — every downstream step fails too. Build can’t build targets that aren’t declared. Tests can’t run on binaries that weren’t built. Coverage can’t cover tests that didn’t run.
One root cause, four red lines. This isn’t unique to the soft-fail pattern, but it’s especially annoying when you’re already doing extra work to discover the failures. You parse annotations, find five failures, and then have to mentally deduplicate them to figure out “oh, it’s just gazelle.”
A smarter system would detect the cascade and only report the root cause. We haven’t built that yet.
But there’s a more immediate question: why is mai_cluster breaking your GPU tests in the first place?
Preventing cascade failures in Bazel CI
Here’s the real scenario. You’re working on GPU tests — mai_kernels, the CUDA stuff. You push a PR, Bazel CI kicks off, and you get this:
ERROR: mai_cluster/src/mai_cluster/python_forward_compat/BUILD.bazel:4:11:
missing input file 'remove_over_py310.py'
→ Build did NOT complete successfully
→ No test targets were found, yet testing was requested
mai_cluster has nothing to do with GPU tests. It’s a cluster management package. It has zero transitive relationship to anything in mai_kernels. But bazel test //... tries to build every target in the repo, and if any target fails to build, the whole run dies. Your GPU tests never even got a chance.
This is the //... problem. It means “everything.” And in a monorepo, “everything” includes a lot of things that have nothing to do with what you’re testing.
There are six ways to fix this, from blunt to surgical.
Option 1: Target specific packages
Stop using //.... Name your packages explicitly.
- name: GPU tests
run: bazel test //mai_kernels/tests/... //rocket/tests/...
This is immune to unrelated breakage — mai_cluster could have its BUILD file replaced with a recipe for sourdough and your GPU tests wouldn’t notice. The downside is obvious: when someone adds a new GPU test package, someone has to remember to add it here. They won’t.
Option 2: --keep_going
- name: GPU tests
run: bazel test //... --test_tag_filters=gpu --keep_going
One flag, one line. Bazel continues past build errors and runs whatever can build. mai_cluster fails to build? Fine, Bazel logs the error and moves on to the targets that did build.
Exit code is still non-zero (Bazel reports the build error), so your CI step still “fails” — but with continue-on-error: true in the soft-fail pattern, that’s already handled.
The problem: //... still attempts to build everything. It just doesn’t stop on failure. If you have 200 broken targets and 5 GPU targets, Bazel wastes time trying to build all 200 before running your 5 tests.
Option 3: bazel query first, then test
# Step 1: Find GPU targets (query doesn't build anything)
TARGETS=$(bazel query 'attr(tags, "gpu", //...)')
# Step 2: Test only those targets
bazel test $TARGETS
bazel query operates on the build graph — it reads BUILD files but doesn’t compile anything. A broken remove_over_py310.py doesn’t matter if you never try to build that target.
Two Bazel invocations means more startup overhead, and the query syntax has its own learning curve. But it’s a clean separation of “find targets” from “build and test targets.”
Option 4: --build_tag_filters
This is the one people miss.
- name: GPU tests
run: bazel test //... --test_tag_filters=gpu --build_tag_filters=gpu
Most teams use --test_tag_filters alone and think they’re done. They are not.
Here’s what --test_tag_filters=gpu does by itself: Bazel builds all targets matched by //..., then only runs tests on targets tagged gpu. Read that again. It builds everything. It just doesn’t test everything.
So mai_cluster — with no gpu tag, no relationship to GPU anything — still gets built. And when its BUILD file is broken, your entire test run dies. The test filter doesn’t save you because the failure happens during the build phase, before any tests run.
--build_tag_filters=gpu tells Bazel: don’t even build targets that don’t match the tag. mai_cluster has no gpu tag? Skip it entirely. Don’t parse its deps, don’t look for its source files, don’t touch it.
The difference:
| Flag combo | Builds | Tests |
|---|---|---|
--test_tag_filters=gpu only |
Everything | GPU only |
--test_tag_filters=gpu --build_tag_filters=gpu |
GPU only | GPU only |
One flag away from “build the world and pray” to “build only what you need.” That’s a big one flag.
Option 5: Belt and suspenders
- name: GPU tests
run: bazel test //... --test_tag_filters=gpu --build_tag_filters=gpu --keep_going
--build_tag_filters prevents the mai_cluster class of failures (unrelated package broke). --keep_going handles the “your own dep broke but other GPU tests are still fine” class. Together, they cover both scenarios.
This is the recommended approach. One line, zero maintenance, no target lists to update, no query step, and it handles both failure modes.
Option 6: Targeted + fallback
For the truly paranoid:
- PR CI: Use explicit package targets (
//mai_kernels/tests/...). Fastest, most isolated. - Nightly CI: Use
//...with--build_tag_filters+--keep_goingas a discovery mechanism. Catches new test packages that weren’t added to the explicit list.
This gives you speed in the hot path (PRs) and coverage in the cold path (nightly). The cost is maintaining two configurations.
OK but which one do I pick?
If you’ve been reading carefully, Options 3 and 4 are the real contenders. Everything else is either too blunt (explicit targets), too permissive (--keep_going alone), or a composition of these two. So let’s put them side by side.
# Option A: --build_tag_filters (simple, fast)
bazel test //... --test_tag_filters=gpu,-integration --build_tag_filters=gpu,-integration --keep_going
# Option B: bazel query (robust, explicit)
TARGETS=$(bazel query 'attr(tags, "gpu", //...)' 2>/dev/null)
if [ -n "$TARGETS" ]; then
bazel test $TARGETS --keep_going
else
echo "No GPU test targets found — skipping"
fi
| Dimension | bazel query then test |
--build_tag_filters |
|---|---|---|
| What it does | Find targets first (no build), then test only those | Build + test in one pass, skip non-matching targets |
| Survives unrelated breakage | ✅ query doesn’t build anything | ✅ skips non-tagged targets |
| Survives GPU dep breakage | ❌ still fails (--keep_going: partial) |
❌ still fails (--keep_going: partial) |
| Speed | Slower — two Bazel invocations, cold start twice | Faster — single invocation |
| “No tests found” handling | ✅ check $TARGETS is empty, exit gracefully |
❌ Bazel errors on zero matches |
| Discovers new tests | ✅ query finds everything tagged gpu |
✅ //... finds everything tagged gpu |
| Readability | Multi-line bash with query piping | One-liner |
| Cache | ❌ two commands, startup cost twice | ✅ single command, single cache load |
The table looks close, but the real differentiator is edge cases. Specifically: what happens when zero GPU test targets exist?
With bazel query, you get an empty string. Check it, print a message, exit 0. Clean. With --build_tag_filters, Bazel hits //..., finds no targets matching the filter, and errors: "No test targets were found, yet testing was requested." Your CI step fails — not because anything is broken, but because there’s nothing to test. In a migration where GPU test coverage is still growing, “nothing to test yet” is a completely valid state. You need your CI to handle it gracefully.
The query approach costs you ~30 seconds of extra Bazel startup time. On a GPU test job that takes 5+ minutes, that’s not material. It’s the CI equivalent of stopping to tie your shoes before a marathon — technically slower, nobody cares.
That said, if you know you’ll always have at least one GPU test target (mature repo, stable coverage), --build_tag_filters is the simpler choice. One line, no bash gymnastics, faster execution. It’s the right default for repos past the “do we even have tests yet?” phase.
What we went with
Option 5. The --build_tag_filters + --keep_going combo in the GPU workflow.
The reasoning was simple: we wanted a fix that didn’t require ongoing maintenance. Explicit target lists rot. Query-based approaches add complexity. But --build_tag_filters=gpu is a declarative statement that survives any future changes to the repo — as long as GPU tests are tagged gpu (which they have to be anyway for the test filter to work), the build filter comes for free.
Adding --keep_going on top was a “why not” decision. It costs nothing and handles edge cases we haven’t hit yet.
The limitations
Let me be honest about what this pattern doesn’t solve:
1. The green lie. The job always shows green, which means people stop looking at it entirely. The soft-fail pattern trades “people are confused by red” for “people ignore green.” Whether that’s better depends on your current failure rate. For us, it was — Bazel CI was failing on most PRs due to migration growing pains, so the red was pure noise.
2. No native advisory check in GitHub. This is the root cause of the whole problem. GitHub checks have two states: required (blocks merge) and not-required (shows red but allows merge). There’s no “advisory” status — no yellow, no grey, no “heads up but don’t panic.” We considered using the neutral conclusion instead of success, which shows grey in the UI, but that’s still not great discoverability.
3. Annotation parsing is non-trivial. You can’t just look at the GitHub UI to know the real status. You need API calls, annotation parsing, and some understanding of the step naming convention. We built a CLI for this, but that only helps people who know the CLI exists.
4. No regression notification. If a merge to main introduces a new Bazel failure, nobody notices — because the nightly run shows green regardless. We don’t have a dashboard tracking the real pass rate over time.
5. Image availability gaps. The GPU job depends on a pre-built Docker image that updates on a 2-hour timer. If Dockerfiles change, GPU tests silently skip until the next build cycle. Not a soft-fail problem per se, but it compounds the “everything looks green but isn’t” issue.
6. Cascade noise. One gazelle failure produces four downstream failures. The annotations don’t distinguish root cause from cascading effect.
Where this goes next
The soft-fail pattern is a bridge. It’s the right answer for “we need CI signal from an incomplete migration without blocking anyone.” But it has an expiration date.
Near-term:
- PR comment bot — post a comment on every PR with the real Bazel status, like other CI bots do. Moves the signal from “hidden in annotations” to “visible in the PR timeline.”
- Smarter cascades — if gazelle fails, skip build/test/coverage. Report the root cause, not the symptoms.
- Bazel CI dashboard — Datadog dashboard tracking real pass/fail trends. Catch regressions on main before they pile up.
Medium-term:
neutralconclusion — use GitHub’sneutralcheck conclusion instead ofsuccess. Shows grey in the UI instead of green. Not perfect, but at least it signals “something to look at” without being red.- Image pre-warming — trigger Docker image builds on Dockerfile changes in PRs, not just on a timer. Eliminates the 2-hour gap.
End state:
- Promote to required. Once Bazel coverage is high enough and the failure rate on main is low, flip the switch. Make it a real gate. The soft-fail pattern is explicitly designed to be removed — it exists so that the transition from “no Bazel CI” to “required Bazel CI” doesn’t have to be a single terrifying leap.
The meta-lesson
Every CI team eventually faces this: you have a new check that catches real issues but also has false positives. If you make it required, you block people on things they can’t fix. If you make it optional, nobody looks at it. If you make it soft-fail, people think everything is fine when it isn’t.
There’s no good answer. There’s only “which failure mode can you tolerate right now?”
For us, “people confused by red checks they can’t fix” was worse than “real failures hidden behind green.” So we chose green, built tooling to surface the truth for people who need it, and accepted that the pattern has a shelf life.
The goal was never permanent soft-fail. The goal is to get to the point where we can make Bazel CI required — and when we do, it’ll be because the check is reliable enough that red actually means “you broke something.” Not “the migration isn’t done yet.”
That’s worth a few months of green lies.