Every monorepo maintainer has the same recurring nightmare: a developer pings you because CI failed on their PR, you look at the logs, and the failing test has nothing to do with their change. It passed yesterday. It’ll pass tomorrow. It just decided not to pass right now.
Flaky tests are a tax on developer trust. Every false failure costs you — the developer’s time investigating, the CI re-run minutes, and worst of all, the slow erosion of confidence in the test suite. When developers stop trusting CI, they stop reading CI results. When they stop reading CI results, real bugs ship.
I recently came across a BazelCon 2025 talk and companion blog post from Uber about how they manage flaky tests at scale. Their numbers are staggering: 5,000+ developers, 2,500+ diffs per day, 10,000+ tests per diff. Their Go monorepo alone has 600K tests with about 1,000 detected as flaky at any given time. Java side: 350K tests, another ~1,000 flaky.
What struck me wasn’t the scale — it was the maturity of their approach. They didn’t build a “rerun flaky tests” button. They built a full lifecycle management system. And a lot of it is borrowable without Uber-level resources.
The System: Testopedia
Uber’s flaky test system is called Testopedia, and the first important thing about it is what it’s not: it’s not a test runner. It doesn’t execute tests. It sits between CI test reporting and CI consumers as a centralized tracking service. It ingests test results, decides what’s flaky, and tells CI what to do about it.
This separation is smart and intentional. Your detection logic shouldn’t live inside your test runner or your CI workflow YAML. It should be a service that observes test results and makes decisions independently. Swap test runners, change CI systems, add new languages — the flakiness tracking still works.
Fully Qualified Names
Every test gets a unique identity called a Fully Qualified Name (FQN), scoped within a “realm”:
golang.unit/src/uber.com/infrastructure/module:TestName
| | |
+- realm +-- codebase path ------------------+-- test name
Realms like golang.unit_test or android.integration_test let you apply different detection rules per language and test type without changing core logic. Go unit tests might have a tight flakiness threshold; GPU integration tests might need a more lenient one. Same system, different knobs.
If you’ve ever tried to answer “is this test flaky?” and gotten confused because the same test name appears in three different contexts, the FQN + realm model is the fix.
Finite State Machine
Each test transitions through well-defined states:
New → Stable (initial pass)
Stable → Unstable (fails in window) → files JIRA ticket
Unstable → Stable (N consecutive passes) → closes JIRA ticket
Unstable → Disabled (owner disables / auto-disable)
Disabled → Deleted (test removed)
This is my favorite part of the design. Tests aren’t just “pass” or “fail” in a single run — they have a tracked lifecycle. When a test transitions to Unstable, a ticket gets filed. When it recovers, the ticket closes. When it stays Unstable too long, it gets disabled.
No ambiguity. No “is this test known flaky?” Slack conversations. The system tells you.
Detection: How They Decide What’s Flaky
The Intentional Asymmetry
Their default analyzer uses a simple but clever rule: one failure in the last X runs marks a test as Unstable. But to recover to Stable, the test must pass N consecutive times.
The asymmetry is the key insight. Easy to become flaky (one failure), hard to become un-flaky (consistent consecutive passes). This matches reality — a single unexpected failure is a real signal, and a handful of passes doesn’t mean the problem is fixed.
If you’ve ever marked a test as “fixed” because it passed three times, only to have it fail again the next day — this asymmetry is designed to prevent exactly that.
Alternative Analyzers
The linear analyzer is the default, but they support pluggable alternatives:
- Percentage-based: For integration tests and GPU tests where some failure rate is expected. “Flaky if failure rate exceeds X% over last N runs.”
- Trend-based: Looks for patterns over time — is the failure rate increasing?
- Custom plugins: Per test type, per realm.
Not all tests are equal. A unit test that fails once in 100 runs is suspicious. A GPU test that fails once in 100 runs might just be hardware jitter. Same system, different rules.
Data Stream Separation
This one’s subtle and important. Uber separates test results into labeled streams:
- Periodic monitoring: Scheduled CI runs, not triggered by any diff
- Regular CI: Diff-triggered test runs
- Pre-land retry: Tests that failed then passed on retry
Each stream feeds into analyzers independently. Why does this matter? Signal quality.
A test that fails in periodic monitoring (no code change at all) is almost certainly flaky — nothing changed, so the failure is non-deterministic by definition. A test that fails in regular CI might be a legitimate failure caused by the diff. A test that fails then passes on retry is the strongest flakiness signal of all.
If you’re lumping all your CI results into one bucket, you’re mixing signals. The same failure means different things depending on how the test was triggered.
Pre-Land Flake Detection
Before a diff lands, tests run. If a test fails once but passes an identical retry, it gets proactively flagged as flaky — before it ever reaches the main branch. Catch it early. Don’t wait for it to break main branch CI.
This is one of the most borrowable ideas in the whole system. Even without a centralized service, you can implement this: run known-fragile tests twice on PR CI, flag fail-then-pass patterns, and track those tests separately.
Mitigation: What Happens After Detection
Three-Tier Response
Once a test is flagged, Uber doesn’t just skip it. Three tiers:
| Tier | Behavior | When |
|---|---|---|
| Critical | Always run, always block | Tests you can’t afford to skip |
| FYI mode | Run but don’t block the diff | Known-flaky — results visible but don’t gate merges |
| Skip | Don’t run (but still build the target) | Persistently flaky, actively being fixed |
The critical nuance: even skipped tests are still built. Only execution is skipped. This catches compilation errors without running the flaky test. You lose test coverage for the flaky behavior, but you don’t lose build validation.
Engineers can also opt-in to running flaky tests on their diffs via tags. Working in an area covered by a known-flaky test and want to see its output? You can — it just doesn’t block everyone else.
Cache Stability
Bazel-specific detail but the principle is universal. When Uber marks a test as flaky, they pass the flaky test list via environment variables — not Bazel rule inputs. If flakiness state were a rule input, updating the flaky test list would invalidate Bazel’s cache for every affected target. The flakiness status changes daily (maybe hourly), but the actual test code hasn’t changed. No reason to invalidate the cache.
General principle: your flaky test metadata should not couple to your build graph. Keep detection data out of your build system’s caching inputs.
Test Case Granularity
This part of the BazelCon talk resonated because it’s a problem I’ve hit directly.
Bazel runs tests at the target level — a target is a binary. But flakiness is often at the test case level. One flaky test case in a target with 50 test cases means you either skip all 50 (losing coverage) or skip none (the flaky case blocks CI). Neither is acceptable.
Uber invested heavily in making rules_go generate proper test.xml reports with individual test case results:
- Individual test case results — not just pass/fail for the whole binary
- Reports even on timeout — previously, a timeout produced no report at all
- Elapsed time per test case — for identifying slow-and-flaky tests
- Failure events on panics — a panic that kills the test binary should still produce a result
- Subtests grouped into test suites — proper hierarchical reporting
The Dynamic Name Problem
Go’s table-driven tests often generate test case names dynamically. When names aren’t deterministic, FQN tracking breaks — you can’t track the stability of a test you can’t consistently identify.
Their fix: report subtests as they finish (streaming) instead of buffering until the end, and sync Go’s test2json from upstream. The kind of detailed plumbing work nobody celebrates but makes the whole system viable.
If you use pytest with parameterized tests, you’ve probably hit the same problem. Make sure your test IDs are deterministic and stable across runs, or your flakiness tracking will be full of ghosts.
Notification and Ownership
Auto-Filing Tickets
When a test transitions to Unstable, Testopedia inserts an event into a message queue. A Cadence/Temporal workflow periodically processes the queue and files a JIRA ticket to the owning team. When the test recovers, the ticket auto-closes.
Key design choice: asynchronous. Detection doesn’t block on ticket filing. The workflow can retry, batch, and deduplicate without affecting CI latency.
Smart Grouping
Five test cases in the same target all go flaky? One ticket, not five. Grouping by build target, regex pattern, or team ownership. Prevents the “JIRA flood” problem where 20 tickets land on a team’s board in an hour saying the same thing.
Per-Realm Customization
Each test realm configures its own ticket type, priority, description templates, grouping rules, and notification frequency. Go unit tests might file P2 bugs; Android integration tests might file P3 tasks with a different template. Same system, different policies.
What I Want to Borrow
Not all of this requires Uber-level investment. Roughly ordered from quick wins to bigger bets.
Quick Wins
1. Data stream separation. Tag your CI test results. At minimum: “PR-triggered”, “scheduled”, and “retry”. Even if you do nothing else with this data today, having the labels makes future analysis possible. This is a CI workflow change, not a new service.
2. Pre-land flake detection. For your most fragile tests, run them twice on PR CI. If a test fails then passes, flag it. A CI workflow that retries known-fragile tests and reports discrepancies is enough.
3. Test case granularity in reporting. Check your test report format. Individual test case results, or just pass/fail for the whole suite? For pytest, make sure your JUnit XML includes individual cases. This is the foundation everything else builds on.
Medium Investment
4. Finite state machine for tests. Track test stability state persistently — even just a database table with (test_id, state, last_transition_date). You don’t need a full Testopedia to get value from knowing which tests are currently Unstable.
5. FQN-based test identity. Normalize test IDs so a parameterized test has the same ID across runs, a renamed test can be tracked through its rename, and you can query “show me all flaky tests in this directory.”
6. Auto-ticket filing. Test Unstable for more than N days? Auto-file a work item to the owning team. Include failure rate, last N failure logs, link to the test. Close it when the test recovers. Even a cron job is valuable.
7. Non-blocking mode for flaky tests. Run known-flaky tests in CI but don’t let them block merge. Report results as informational. Preserves visibility without blocking velocity.
Larger Bets
8. Centralized flaky test service. A “Testopedia-lite” that ingests results, tracks state, serves flakiness decisions to CI. Real infrastructure — service, database, API — but the foundation for everything above.
9. Sliding window analyzer. The asymmetric detection rule: 1 failure in last N runs = unstable, M consecutive passes = stable. Configurable per test type.
10. Cache-stable flaky test skipping. Make sure flaky test metadata doesn’t invalidate the cache. Pass flaky test lists via env vars or external config, not build inputs.
The Core Insight
The biggest takeaway is philosophical, not technical: flaky tests are a data problem, not just a code problem.
The natural instinct is to treat each flaky test as an individual bug — find the test, fix the non-determinism, move on. That works when you have 10 flaky tests. It doesn’t work when you have 1,000. At scale, you need a system: centralized detection, persistent state tracking, automated mitigation, and ownership-driven resolution.
The data framing also changes how you prioritize. Fixing one flaky test is a code task. Building a system that detects and mitigates flaky tests across the whole monorepo is an infrastructure investment. The first helps one team today. The second helps every team forever.
A few things worth pinning:
- Granularity matters. Test-case-level tracking enables precise mitigation. Target-level tracking forces you to skip-or-run entire suites.
- Detection and mitigation are separate concerns. Deciding a test is flaky is a different problem from deciding what to do about it.
- Ownership drives resolution. Auto-filing tickets with good context is what actually gets flaky tests fixed. Detection without notification is just a dashboard nobody checks.
- Separate your data streams. Periodic monitoring, PR CI, and retry CI are different signals. Don’t mix them.