We Built an AI CI Fixer — Then Discovered It Was Broken

2026/03/05

BUILDai-toolingcidevexmonorepo

Two months ago we shipped an AI-powered CI failure fixer. Claude Code in CI — analyze test failures, classify them, push fix branches automatically. The dream.

In 60 days of operation, it created exactly one auto-fix branch.

One.

We assumed the model was the bottleneck. Turns out we never gave it the keys to the car.

The setup

Quick context. We maintain a large Python ML monorepo — about a hundred packages, GPU tests, CPU tests across multiple clusters, the whole thing. CI breaks a lot. We averaged 12.6 failure alerts per day in the investigation channel, 85% of which got ignored.

The existing investigation pipeline was already pretty good. It detects main branch failures, classifies them (flaky vs regression vs pre-existing), runs git bisection to find blame commits, creates revert PRs, and posts to Slack. I wrote about that system a few weeks ago.

The problem: even with a revert PR and a blame commit identified, the original author still had to spend 30-60 minutes reading CI logs to write a proper fix. And the team’s response to skip PRs was… not great. 59 skip PRs and 10 revert PRs in 60 days. 19 skip PRs stuck open, forgotten. One person handling 69% of investigation PR merges — a single point of failure who presumably has other things they’d like to do with their life.

So we added the AI step. After bisection finds a blame commit, dispatch Claude Code to analyze the failure, classify it, and — for simple errors — create a fix branch. anthropics/claude-code-action@v1, Vertex AI backend, Claude Sonnet 4.5.

Ship it. Watch it work.

Watch it not work.

The gap

I started investigating why the AI step was producing analysis comments but never creating fix branches. The first thing I did was compare what the CI Claude actually gets versus what I get when I debug failures locally with Claude Code.

The gap is comical.

CI Claude (the ai_analysis job in investigate_failures.yml):

Local Claude Code (what I use to debug the same failures):

It’s like comparing a blindfolded intern with a flashlight to a senior engineer with their entire development environment. Same underlying intelligence, wildly different tooling.

The bug

Here’s where it gets good. The prompt in the ai_analysis job tells Claude:

Create a new branch auto-fix/{test_file_name} from current HEAD. Make the fix. Commit with message: fix: {brief description}. Push the branch to origin.

Great instruction. Clear, actionable. One problem.

The allowedTools for that job:

--allowedTools 'Read,Write,Edit,Bash(git diff*),Bash(git log*),Bash(git show*),Grep,Glob'

Read that list carefully. git diff, git log, git show. All read-only operations.

There is no git branch. No git checkout. No git add. No git commit. No git push.

The prompt says “create a branch and push it.” The tool configuration says “you may look at things.” The agent is told to drive to the store but the car has no steering wheel.

Now here’s the kicker — there’s a separate workflow file, ai_fix_failure.yml, that was the original standalone AI fixer. Its tools:

--allowedTools 'Read,Write,Edit,Bash(git diff*),Bash(git log*),Bash(git show*),Bash(git branch*),Bash(git checkout*),Bash(git add*),Bash(git commit*),Bash(git push*),Grep,Glob'

All the git write operations. Branch, checkout, add, commit, push. This workflow can fix things.

But it’s dead code. The investigation pipeline doesn’t dispatch to it anymore — it runs the ai_analysis job inline instead. The workflow with the right tools is never triggered. The workflow that runs has the wrong tools.

This is why exactly one auto-fix branch exists. Probably from an early manual test of ai_fix_failure.yml before the integration changed.

The classification problem

Even if the tools were right, there’s a second gate. The prompt has a conservative classification:

AUTO-FIX allowed:

ANALYSIS ONLY (no fix attempt):

That last bullet: “when in doubt, analyze only.”

So the system has two layers of conservatism. The tool permissions physically prevent fixes, and the classification prompt defaults to “don’t try” for anything ambiguous. Belt, suspenders, and the pants are sewn shut.

In a monorepo with GPU tests, CUDA errors, and distributed training — a lot of failures hit the “analysis only” bucket. The auto-fix category is basically “typos and missing imports.” Which, to be fair, is a reasonable starting point. But combined with the tools lockout, it means the system almost never even attempts a fix, and when it does attempt one, it can’t ship it.

What an evaluation would look like

Before changing anything, I wanted to understand whether the bottleneck is the model, the classification thresholds, or the context. So I designed an offline replay evaluation — not implemented yet, but the shape of the thing.

Data source: The pipeline already saves ai-analysis-requests/*.json as workflow artifacts. Each one has test_ids, package, blame_sha, error_text, committer, revert_pr_number. That’s the input.

Ground truth: Cross-reference with what actually fixed each failure. Did someone merge the revert PR? Write a fix-forward? Was it a skip PR that’s still stuck open? Did it self-resolve?

Replay: Feed the same error context through the model (offline, not in CI) and grade the output:

Grade Meaning
A Correct fix — matches what the human actually did
B Partial fix — right direction, incomplete
C Correct analysis — identified root cause, didn’t attempt fix
D Wrong analysis — misidentified the problem
F Wrong fix — would introduce a new bug

The diagnostic question: for failures where the model gets a C (“correct analysis, didn’t fix”), is it because the classification said “don’t try” or because the model genuinely couldn’t figure out a fix?

The highest-value experiment is context ablation. Same 30-50 failures, four context levels:

  1. Minimal — just error text and test file
  2. Standard — error text + blame diff + test file (what CI Claude currently gets)
  3. Rich — standard + historical failure patterns for this package
  4. Full — rich + package dependency graph + previous fix PRs for similar errors

If the model jumps from D/C grades to A/B when you add historical patterns, then the bottleneck is context, not capability. Build a knowledge base. If it stays at C regardless, the classification thresholds are too conservative. If it produces F-grade fixes even with full context, the model genuinely can’t do this yet.

Feeding two different Claudes

Assuming context matters (it almost certainly does), the question is how to deliver failure knowledge to two very different consumers: CI Claude (constrained, read-only, cold start) and local Claude Code (full-featured, persistent memory).

The architecture is two layers.

Layer 1: Shared knowledge base. A YAML file checked into the repo — ci/knowledge/failure_patterns.yml. Human-readable, grep-able, git-tracked, reviewable in PRs.

patterns:
  - id: torch-import-after-upgrade
    error_type: ImportError
    signature: "cannot import name .* from 'mai_layers'"
    packages: [mai_layers, mai_trainer]
    typical_fix: >
      Check blame commit for torch version bump.
      Update import path in mai_layers/__init__.py.
    confidence: high

  - id: fixture-missing-after-conftest-move
    error_type: FixtureError
    signature: "fixture .* not found"
    packages: ["*"]
    typical_fix: >
      conftest.py was likely moved or renamed.
      Check git log for conftest changes in the package.
    confidence: medium

This gets populated from Slack investigation threads, merged fix PRs, patterns that show up repeatedly. It’s a knowledge base that humans maintain and the AI consumes.

Layer 2a: CI delivery. A Python pre-filter runs at prompt build time, before Claude sees anything. It scores each pattern against the current failure:

signature regex match  → +3
error_type match       → +2
package match          → +1
minimum score to include: 2

Top 3-5 matches get injected into the prompt as additional context. CI Claude sees it as pre-loaded knowledge, zero tool calls needed. Maybe 2-5KB of extra prompt, well within budget.

Layer 2b: Local delivery. When I’m debugging locally, I just grep the YAML file, then enrich with Slack threads, Datadog dashboards, git history. The same knowledge base, but with a much richer access pattern.

The elegant thing: the YAML file is the single source of truth. CI gets a filtered view. Local gets the full thing plus live context. Both are reading from the same knowledge.

The prompt structure

For the record, here’s what the CI agent actually receives — broken into logical sections:

  1. System context — “You are investigating CI test failure(s) in a Python ML monorepo.”
  2. Failure context — Test IDs, package name, blame commit SHA, committer, revert PR number, error text (truncated to 8KB)
  3. Strategy directive — The investigate → classify → fix-or-analyze three-step prompt
  4. Constraints — “Don’t run tests, keep fixes minimal, prefer analysis over risky fixes”

What’s missing:

The agent is doing root cause analysis with a partial stack trace and the blame diff. It’s like diagnosing a car problem by listening to the engine noise while someone describes the dashboard warning light over the phone. You can sometimes get it right. But you’re working way harder than you need to.

The punchline

The model might be fine. We genuinely don’t know, because we’ve never tested it under fair conditions. What we do know:

The fix isn’t a better model. It’s fixing the plumbing.

Unlock the git write tools in the ai_analysis job (or rewire it to dispatch to ai_fix_failure.yml which already has them). Lower the classification threshold — or better, let the model attempt fixes for more error types but behind a human review gate. Inject historical failure patterns into the prompt. Build the evaluation pipeline so we can actually measure accuracy and tune thresholds.

There’s a broader lesson here, and it’s embarrassingly obvious in retrospect: the gap between what you think your AI system is doing and what it’s actually doing can be enormous. We had a working prompt, a capable model, a clean integration architecture. But the allowedTools line was wrong and nobody noticed for two months because the system was producing something — analysis comments got posted, Slack notifications fired, the workflow showed green checkmarks. It looked like it was working. It was just working as a very expensive rubber duck.

The hardest bugs aren’t the ones that produce errors. They’re the ones that produce slightly less output than you expected, in a system where “less output” looks normal.