Adding AI Failure Analysis to Our CI Pipeline (Without Letting It Break Anything)

When a CI test breaks on main branch, someone has to figure out why. In a monorepo with dozens of contributors, that someone is usually a maintainer staring at log output at 10 PM, trying to determine whether an ImportError is a missing dependency or a circular import three levels deep.

We already had automation for the first half of this problem. Our investigate_failures.yml workflow detects main branch failures, classifies them, and — for regressions — runs git bisection to find the blame commit and creates a revert PR. That part works well. But the revert PR lands with no context about why the test broke, which means the original author still has to spend 30-60 minutes reading CI logs before they can write a proper fix.

So we added an AI analysis step. After bisection finds the blame commit, the pipeline now dispatches a separate workflow that uses Claude Code to analyze the failure, classify it, and — for a narrow set of simple errors — attempt a fix. Here’s how the full investigation pipeline works, how we built the AI layer on top, and what we learned.

The Full Investigation Pipeline

Before getting into the AI parts, it helps to understand the complete investigation system. The AI step is just the final leaf of a much larger decision tree. Most CI failures never reach it — and that’s by design.

The pipeline triggers automatically when “Main Required CI” fails, or manually via workflow_dispatch with a run ID. It runs four parallel investigation jobs — GPU, CPU shard, CPU shard big, and CPU condor — all calling the same core Python logic.

Step Zero: The System-Wide Guard

Before investigating individual failures, the pipeline groups all test failures by (package, shard). If more than 3 package/shard combinations have failures, it aborts entirely. No Slack notifications, no PRs, no investigation.

Why? When that many things break simultaneously, it’s almost certainly an infrastructure issue — a container build failure, a dependency resolution problem, a runner going down. Individual test investigation would be noise. Better to log it and let a human look at the systemic problem.

This is a blunt heuristic, but it prevents the bot from spam-creating dozens of skip PRs and revert PRs when the real problem is that Docker Hub rate-limited your image pulls.

The Decision Tree

For each (package, shard) group that passes the system-wide guard, the pipeline runs a three-phase investigation:

investigate_failure()
│
├─ Find baseline (last green CI run via GitHub API)
│   └─ No baseline found? → Slack "SKIPPED", stop
│
├─ PHASE 1: Run test on current HEAD
│   │
│   └─ PASS? → run 3 more times → classify as FLAKY
│       → Create skip PR (@pytest.mark.skip)
│       → Create Linear issue (under CICD-94)
│       → Notify Slack
│       → STOP (no bisection needed)
│
├─ PHASE 2: Run test on baseline commit
│   │
│   ├─ FAIL → run 3 more times
│   │   │
│   │   ├─ 0 of 4 pass → BASELINE BROKEN
│   │   │   → Create skip PR
│   │   │   → Create Linear issue
│   │   │   → Notify Slack
│   │   │   → STOP
│   │   │
│   │   └─ >0 of 4 pass → BASELINE FLAKY
│   │       → Create skip PR
│   │       → Create Linear issue
│   │       → Notify Slack
│   │       → STOP
│   │
│   └─ Setup error → BASELINE SETUP FAILED
│       → Notify Slack (no PR — can't proceed)
│       → STOP
│
└─ PHASE 3: Binary search (git bisect)
    │
    │  For each mid commit:
    │  - Creates git worktree at that commit
    │  - Installs deps (uv sync --frozen)
    │  - Checks test exists via AST parsing
    │  - Runs pytest
    │  - Errors treated as BAD (conservative)
    │
    └─ Found blame commit → REGRESSION
        → Create revert PR (git revert)
        → Create Linear issue
        → Notify Slack (blame commit + author)
        → Trigger AI analysis workflow ← the new part

The critical thing: the pipeline classifies failures before taking action. A test that passes on the current commit is flaky — no need to bisect. A test that also fails on the known-good baseline is pre-existing — bisection would be meaningless. Only confirmed regressions (fails on HEAD, passes on baseline) enter the expensive bisection phase.

Actions by Classification

Every PR — skip or revert — goes through duplicate detection first. If an open PR with a matching title already exists, the pipeline skips creation and posts to Slack that it found an existing one. This prevents duplicate PRs when the investigation reruns.

All PRs get auto-labeled fix-lint, run-gpu-tests, skip-code-review to expedite the CI/review cycle.

The Classification Gate

This is the part I’m most opinionated about: the AI doesn’t get to decide what to do. We decide what the AI is allowed to do, based on the error type.

Every failure that reaches the AI step gets classified into one of two categories:

The classification is intentionally conservative. An ImportError caused by a missing import statement is a well-defined problem with a well-defined fix — the AI can handle this reliably. A CUDA out-of-memory error might require changing batch sizes, restructuring a training loop, or upgrading to a different GPU SKU. That’s not something an AI should be auto-fixing in CI.

The line we drew: if a human would fix it by looking at the error message and making a mechanical code change, auto-fix is appropriate. If a human would need to understand the system’s behavior to fix it, analysis-only.

Even in auto-fix mode, the AI creates a branch — it doesn’t merge anything. A human still reviews the fix PR. The automation saves the 30-60 minutes of log reading and root cause analysis, not the code review.

Architecture

The implementation is two workflows connected by a workflow_dispatch event:

investigate_failures.yml (existing, modified)
  └── investigate_failure()
        └── trigger_ai_analysis()  ──dispatch──>  ai_fix_failure.yml (new)
                                                    ├── Classify error type
                                                    ├── Claude Code analysis
                                                    ├── (if AUTO-FIX) create fix branch
                                                    └── Post analysis on revert PR

The trigger_ai_analysis() function is best-effort and non-blocking. If the dispatch fails — API error, rate limit, whatever — the investigation pipeline continues normally. The revert PR still gets created. The AI analysis is additive; its absence never degrades the existing workflow.

This was a non-negotiable design constraint. CI investigation is a critical path. Adding AI to it should not make it less reliable.

Claude Code via Vertex AI

The AI step uses anthropics/claude-code-action, which runs Claude Code as a GitHub Action. We use Vertex AI as the backend, authenticated via GCP Workload Identity Federation. No API keys stored in secrets, no tokens to rotate.

Claude Sonnet 4.5, max 50 turns, with scoped tool access: read-only git operations and file read/edit. It can’t push, can’t merge, can’t modify workflow files. It operates in a sandbox with just enough access to understand the codebase and propose changes.

If the first analysis attempt fails (model error, timeout, transient issue), the workflow retries once after a 30-second delay. Two failures and it gives up — again, best-effort.

Helper Functions

Three small utilities support the classification and analysis:

• check_test_exists() — Uses AST parsing to verify a test function actually exists in the codebase before telling the AI to look at it. Prevents the AI from analyzing phantom tests that were renamed or deleted. Also used during bisection — if a test doesn’t exist at a historical commit, that commit is treated as “pass” rather than a false failure.
• get_package_path() — Resolves a test path to its monorepo package, so the AI gets the right context about which package broke.
• truncate_error() — CI logs can be enormous. This truncates error output to a reasonable size while preserving the useful parts (the actual error message, not 500 lines of stack frames from third-party libraries).

Testing

Testing CI workflows is its own special kind of challenge. You can’t unit test a GitHub Actions workflow in the traditional sense — it runs on GitHub’s infrastructure, triggered by GitHub events, with access to GitHub APIs. But you can test the pieces.

Unit Tests

The Python helper functions (check_test_exists, get_package_path, truncate_error, error classification logic) have 620 lines of unit tests. These run in regular CI, no GPU required, no GitHub API access needed.

E2E Testing

We tested ai_fix_failure.yml independently using gh workflow run with synthetic inputs. Three scenarios:

This was possible because we designed ai_fix_failure.yml to accept all its context as workflow_dispatch inputs — it doesn’t depend on being triggered by the investigation pipeline specifically. You can run it manually with test data.

The Full Pipeline Test

After merging, we triggered the complete flow on a real CI failure: investigate workflow detects failure, runs bisection, finds blame, creates revert PR, dispatches AI analysis. The AI workflow picks up, classifies the error, runs analysis, and posts results on the revert PR.

This is the test you can’t run before merge, because investigate_failures.yml checks out its scripts from origin/main. More on that below.

Interesting Problems

The Chicken-and-Egg Input Problem

GitHub Actions workflow_dispatch inputs are read from the workflow file on the default branch (main), not from the branch that defines the workflow. This means if you add a new input to ai_fix_failure.yml on a feature branch, you can’t test dispatching with that input until after you merge to main.

We hit this with the ci_branch input, which tells the AI workflow which branch contains the CI scripts to check out. The input didn’t exist on main, so dispatching from our feature branch would fail with an “unexpected input” error.

The workaround: merge the workflow file first with a note in the input description explaining the bootstrap constraint, then test the input in a follow-up. Not elegant, but it’s a fundamental GitHub Actions limitation.

The origin/main Checkout Pattern

Our investigation pipeline checks out its CI scripts from origin/main, not from the commit being investigated. This is a stability pattern — you don’t want a broken commit’s CI scripts to be the ones investigating the breakage.

But it means trigger_ai_analysis() only gets exercised by the full pipeline after it exists on main. During development, you can test the AI workflow independently (as we did), but the dispatch integration is untestable until merge.

We accepted this tradeoff. The dispatch call is simple — it’s a gh workflow run with input parameters — and we tested both sides independently. The integration point is small enough that visual inspection of the dispatch code was sufficient.

Enhancing the Existing Pipeline

While adding the AI dispatch, we also improved investigate_failures.yml in ways that had been on the backlog:

• Manual workflow_dispatch trigger — Previously you could only investigate failures that CI detected automatically. Now maintainers can manually trigger investigation for any test, which is useful for intermittent failures that don’t trigger the automatic path.
• Jobs pagination — The GitHub API returns max 100 jobs per page. Tests in our monorepo regularly exceed this. We added pagination so bisection actually finds the failing test instead of silently missing it.
• Artifact naming fixes — Investigation artifacts had naming collisions when multiple investigations ran concurrently.
• Refactored dispatch handling — The workflow now cleanly handles both workflow_run (automatic) and workflow_dispatch (manual) triggers with shared investigation logic.

These weren’t AI-related changes, but they were necessary for the AI integration to work correctly, and they improved the pipeline independently.

What This Does Not Do

Worth being explicit about the boundaries:

• Does not auto-merge fixes. Auto-fix creates a branch. A human reviews and merges.
• Does not replace the revert PR. The revert still happens immediately. The AI analysis is supplementary — it helps the original author write a proper fix faster, but the revert keeps main green now.
• Does not analyze flaky tests. Flaky failures are filtered out before the AI step. Flaky test analysis is a different problem with different solutions.
• Does not handle complex bugs. CUDA errors, race conditions, logic bugs — these get a structured analysis comment, not an attempted fix. The AI tells you what it thinks happened; you decide what to do.
• Does not block the pipeline. If the AI step fails entirely, nothing else is affected. The revert PR still exists, the blame commit is still identified, developers are still notified.

Results So Far

It’s early. The pipeline has been live for a short time and most main branch failures are flaky (which skip the AI path entirely). But the first real activation — an actual regression caught by bisection — produced a useful analysis comment on the revert PR that correctly identified the root cause and the specific lines in the blame commit responsible.

The value is clearest for the “3 AM revert PR” scenario: a test breaks overnight, the pipeline creates a revert PR and an analysis comment. When the original author checks in the morning, instead of starting from scratch with raw CI logs, they have a structured breakdown of what went wrong and why. That turns a 30-60 minute investigation into a 5-minute read-and-confirm.

What’s Missing: Lessons from Uber’s Testopedia

After writing about Uber’s flaky test system, I keep noticing the gaps in our own pipeline. We built a good reactive system — something breaks, we investigate, we create a PR. But compared to Testopedia, we’re missing the proactive layer entirely.

Here’s what bugs me most.

No Persistent Test State

This is the biggest one. Our pipeline treats every failure as if it’s the first time it’s ever seen that test. There’s no memory. A test that flakes every Tuesday and gets auto-skipped every Tuesday generates a new Linear issue and a new skip PR every time.

Uber’s system tracks each test through a state machine — New → Stable → Unstable → Disabled → Deleted. When a test transitions to Unstable, a ticket gets filed. When it recovers, the ticket closes. When it stays Unstable too long, it gets disabled automatically.

We don’t have this. We have “did the test fail right now, yes or no.” That’s phase-one thinking.

The fix isn’t complicated in concept — a database table with (test_id, state, last_transition_date, failure_count) would get us most of the way there. But we haven’t built it yet, so the bot happily creates duplicate work for the same chronically flaky test.

No Data Stream Separation

Uber separates test results into streams: periodic monitoring (no code change), PR CI, and pre-land retries. Each stream feeds into different analyzers with different thresholds.

We don’t do this at all. A failure is a failure — we don’t distinguish between “failed during scheduled monitoring” and “failed on a PR diff.” This means we can’t answer basic questions like: is this test flaky independent of code changes (environment issue), or does it only flake when certain files change (interaction issue)?

The scheduled monitoring stream is the killer feature here. Running tests with no code change gives you the purest flakiness signal possible. If a test fails on a no-diff run, it’s non-deterministic by definition. We run our CI on every push to main, but we don’t run periodic no-change builds specifically for flake detection.

No Pre-Land Detection

Our entire investigation pipeline operates post-merge. By the time we detect a flaky test, it’s already on main. It’s already failed someone’s CI run. Someone already wasted time looking at it.

Uber catches flakes pre-land: if a test fails once but passes an identical retry during PR CI, it gets proactively flagged. The flake never reaches main in the first place.

This one’s actually pretty implementable even without a centralized service. Retry a small set of known-fragile tests on PR CI. Track fail-then-pass patterns. Flag them before merge.

No Tiered Response

Our mitigation is binary: a test is either running normally or skipped entirely via @pytest.mark.skip. There’s no middle ground.

Uber has three tiers: Critical (always run, always block), FYI (run but don’t block), and Skip (don’t run, but still build). The FYI mode is the one I want — known-flaky tests still execute, their results are visible, but they don’t gate merges. You keep the signal without the noise.

The “still build even when skipped” detail matters too. A skipped test that compiles correctly catches import errors and refactoring breakage without running the flaky execution. We lose that entirely when we @pytest.mark.skip.

No Feedback Loop on AI Analysis

Right now we have no systematic way to track whether the AI’s analysis was correct or helpful. The comment gets posted on the revert PR and… that’s it. No thumbs-up/thumbs-down. No tracking of “did the developer use this analysis” vs “did they ignore it.” No measurement of accuracy over time.

Without this, we can’t tune the classification gate. We can’t answer “should TypeError be in the AUTO-FIX category?” because we don’t know how well the current AUTO-FIX category is performing.

What I’d Do Differently

Start with analysis-only. We built both auto-fix and analysis-only from the start. In hindsight, I would’ve shipped analysis-only first and added auto-fix after validating that the analyses were consistently useful. The classification gate adds complexity, and the analysis comments are where most of the value is.

Invest more in error truncation. CI logs are messy. Stack traces from deep dependency chains, repeated warning lines, ANSI color codes, interleaved stdout/stderr. Our truncate_error() function is functional but basic. Better log preprocessing would make the AI’s analysis more reliable.

Build test state tracking from day one. The investigation pipeline would be dramatically better with even basic persistent state. Knowing “this test has flaked 12 times in the last month” changes how you handle it — that’s not a skip-and-file-ticket problem, that’s a disable-and-page-the-owner problem. We went reactive-first because it was faster to ship, but the lack of state is now the system’s biggest limitation.

Add scheduled no-change CI runs. It’s the cheapest way to build a flakiness signal. No new infrastructure — just a cron-triggered workflow that runs the test suite against main with no code changes. Failures are flakes by definition. We should have been collecting this data from the start.

The Broader Pattern

The interesting thing about this project isn’t the AI part — it’s the integration pattern. The AI is a tool slotted into an existing automation pipeline, with clear boundaries on what it can and can’t do, non-blocking execution, and human review before any code changes land.

This is the opposite of the “AI agent that autonomously manages your CI” pitch. We didn’t give an AI agent access to our CI system and ask it to figure things out. We built a specific, scoped integration point: after bisection identifies a blame commit, analyze the failure. That’s it.

The conservatism is the feature. An AI that sometimes produces a wrong analysis comment is fine — a human reads it and moves on. An AI that sometimes auto-merges a wrong fix is a production incident. By keeping the blast radius small (branch creation, not merging; comments, not actions), the failure mode is “unhelpful comment” rather than “broken main branch.”

But I also see the limitation now. We optimized for reacting to failures — detect, classify, bisect, revert, analyze. What we’re missing is the proactive layer: tracking test health over time, catching flakes before they hit main, graduating response severity based on history. The reactive system is table stakes. The proactive system is what actually improves test suite reliability over time.

If you’re building something similar, start where we started — the reactive pipeline is genuinely useful and shippable in weeks. But plan for the stateful layer from the beginning. Even just a (test_id, state, last_failure_date) table changes everything downstream. The hardest part isn’t building the state tracking — it’s retrofitting it onto a system that was designed without it.