There’s a progression in CI maturity that nobody talks about. Stage one: your CI tells you what failed. Stage two: it tells you who broke it. Stage three: it tells you why it broke and how to fix it.
Most teams plateau at stage two. We just reached stage three — for every failure type, not just regressions.
Where we were
Our investigation pipeline (investigate_failures.yml + investigate.py) already did a lot. It bisected test failures, found blame commits, created revert or skip PRs, and posted to Slack. When bisection found a specific commit, it handed the error and diff to Claude for root cause analysis. Solid.
But that AI analysis only triggered for one failure type: blame. The pipeline handles three:
| Type | What it means | Had AI analysis? |
|---|---|---|
| Blame | Bisection found the commit that broke tests | Yes |
| Flaky | Test passes intermittently on the same commit | No |
| Infra | Job fails before tests even run (setup, deps, env) | No |
Flaky and infra failures got Slack notifications and skip PRs, but no explanation. A message saying “test_training is flaky” without telling you why — is it a race condition? Shared state? Timing sensitivity? You’d still have to dig through logs yourself.
The team lead asked a reasonable question: why not analyze all three?
The taxonomy
Each failure type needs a fundamentally different analysis approach. You can’t just throw the same prompt at all of them.
Blame is the original path — Claude reads the blame diff, classifies the failure (breaking change, missing dependency, test assumption violation), and either attempts an auto-fix or writes a structured analysis. This already existed.
Flaky is new and more subtle. Claude reads the test code itself — not a diff, because there’s no blame commit. It identifies the flakiness pattern:
- Race condition (parallel execution, no synchronization)
- Shared state (tests polluting each other)
- Timing sensitivity (sleeps, timeouts, system clock)
- Non-deterministic ordering (dict iteration, set ordering)
- Resource leaks (file handles, connections, GPU memory)
- External dependencies (network calls, filesystem assumptions)
Each pattern gets a different fix suggestion. And each analysis answers: “Quick win? Yes or No” — because some flaky tests need a one-line @pytest.mark.timeout and others need a redesign.
Infra is the most mechanical but arguably the most useful. When a job fails at the setup step — before any test code runs — Claude categorizes it:
- Dependency install failure
- Network/registry issue (ACR login, pip timeout)
- Runner environment problem
- Resource exhaustion
- Configuration error
And critically: “Would a retry fix this?” Because if it’s a transient ACR DNS blip, yes. If it’s a broken base image, no. That single bit of information saves a human from clicking into the job, scrolling through logs, and making the same judgment call.
The wiring
The implementation had three insertion points in investigate.py — each one a spot where the pipeline already knew the failure type but wasn’t asking for analysis.
Flaky (current commit): After create_skip_pr() for tests that pass intermittently. We now capture the skip PR number and write an AI analysis request with flaky metadata — pass rate, error text, the works.
Flaky (baseline): Same pattern, different trigger — when baseline tests themselves are flaky or broken.
Infra: After the Slack notification in investigate_failures_from_artifacts(). This one fires regardless of whether Slack is configured, because analysis and notification are separate concerns. You might not have a Slack token in your environment, but you still want the AI to tell you what went wrong.
A small helper — _build_flaky_error_text() — composites the flakiness metadata (pass rate, failure reason) with the actual test error output into a single string. The prompt is smart enough to parse the structure without needing separate schema fields. Pragmatic over principled.
The prompt architecture
The original prompt was a linear flow: investigate, classify, fix-or-analyze. That worked when there was only one path.
The new prompt uses routing:
Step 1: Determine analysis type from context (BLAME / FLAKY / INFRA)
Step 2a: Blame analysis path → read diff → classify → fix or analyze
Step 2b: Flaky analysis path → read test code → identify pattern → suggest fix
Step 2c: Infra analysis path → read job/step names → categorize → retry assessment
Each path has its own classification categories and output format. The analysis_type field in the JSON request tells Claude which path to take — no guessing.
On the output side, PR commenting got smarter too. Blame failures have revert PRs, flaky failures have skip PRs, infra failures typically have neither. The posting logic checks both revert_pr_number and skip_pr_number to find the right place to comment, and varies the header — “AI Failure Analysis” vs “AI Flakiness Analysis” vs “AI Infrastructure Analysis.” Small thing, but it matters for scannability when you’re triaging a dozen failures at 9 AM.
What we deleted
We also killed the old ai_fix_failure.yml workflow — 367 lines of the v1 approach that used GitHub workflow dispatch. That was the original “cold DM to working pipeline” implementation from a month ago. It worked, but the artifact-based handoff pattern (investigate.py writes JSON, ai_analysis job picks it up) is cleaner and doesn’t require cross-workflow dispatch permissions.
Along with it went trigger_ai_analysis() and truncate_error() — dead code from the dispatch era. Always satisfying to delete more lines than you add.
Design decisions worth noting
analysis_type defaults to "blame" — old JSON files without the field still work. Backward compatibility for free.
Flaky metadata goes into error_text rather than new schema fields. We considered adding pass_rate and flaky_reason as top-level fields, but _build_flaky_error_text() composites them into a rich string that the prompt parses fine. One less schema migration.
Infra fires without Slack. The original code only did things inside if slack_token: blocks. But AI analysis shouldn’t depend on notification infrastructure. These are separate concerns — you want the analysis artifact even if nobody’s listening on Slack yet.
The progression
Here’s what the failure experience looks like now:
Before: CI run fails. Slack says “test_training is flaky.” You open the job, scroll through logs, read test code, think for five minutes, realize it’s a race condition in the teardown. Twenty minutes gone.
After: CI run fails. Slack says “test_training is flaky.” Skip PR gets a comment: “Flakiness pattern: shared state — test modifies module-level config dict without resetting. Quick win: Yes — add fixture to snapshot and restore config.” You fix it in two minutes.
That delta — twenty minutes to two — multiplied across a team of fifty engineers hitting CI failures daily, is the whole pitch for AI in CI. Not replacing humans. Just frontloading the diagnosis so engineers start from understanding instead of archaeology.
What’s actually hard
The implementation was straightforward — five tasks, thirty tests passing, clean lint. The hard part was the taxonomy.
Deciding that flaky tests have exactly six patterns, that infra failures have exactly five categories, that “would a retry fix this?” is the right discriminating question for infra — those are judgment calls that determine whether the AI output is actually useful or just noise.
Get the categories wrong and Claude produces vague analysis that engineers ignore. Get them right and it becomes the first thing people check.
We’ll find out which one this is when it hits production.