The Last Mile Problem in AI CI Fix Loops

We built an AI that can analyze CI failures, identify root causes, and write fixes. It pushes a branch with the fix. And then… nothing happens. No PR. No CI run. No iteration. The branch just sits there, orphaned, like a letter you wrote but never mailed.

This is a post about building CI automation in layers, and how each layer you add reveals the next bottleneck. The glamorous part — teaching an AI to understand test failures — was the easy part. The hard part is the plumbing.

Where we were

Previously, I wrote about our investigate_failures pipeline. Quick recap: main CI fails, the bot auto-bisects to find the blame commit, creates a revert PR, and dispatches Claude Code to analyze the failure. For simple errors (ImportError, NameError, SyntaxError), it even writes a fix and pushes a branch.

That system has been running for about two months. Here’s what the data looks like.

The numbers

I went through 60 days of Slack alerts, GitHub events, and the support channel to understand what’s actually happening.

Pre-automation (Oct–Nov 2025):

Alert fatigue was the dominant failure mode. Over twelve alerts a day, most of them noise. One person absorbed the cognitive load of triaging nearly all of them. That’s not a process — it’s a single point of failure who happens to be very patient.

Post-automation (Jan–Mar 2026):

So the bot creates PRs. That works. But it traded one single point of failure (Jay triaging) for another (Yipu merging). Nineteen skip PRs are sitting open — nobody ever looked at them. And the AI auto-fix feature? One branch. In sixty days. The feature is basically decorative.

The voice of the customer

From our #support-ci-cd channel, over the same period:

“This slows down development quite a bit since we have to babysit PRs for half a day”

“I had to manually re-run the timed-out rocket CI tests like 20 times”

“Reverting it might be harder than fixing it”

That last one is interesting. Merge commits can’t be auto-reverted cleanly. When main has moved on, reverting introduces conflicts with subsequent work. The team often ends up choosing “fix forward” over “revert” — which is exactly the scenario where an AI fix loop would be most valuable.

Other patterns: 32 force-merge events as an escape valve for stuck PRs. Explicit asks from developers for better bisect automation, better failure visibility, faster feedback. The automation helped, but it created new bottlenecks.

Why the AI doesn’t fix anything

Here’s the punchline. The reason only one auto-fix branch exists in sixty days is a combination of two problems:

Problem 1: The classification gate is too conservative.

The prompt tells Claude to only attempt fixes for a narrow set of “simple” errors — ImportError, NameError, SyntaxError, simple assertion failures. Everything else gets classified as ANALYSIS_ONLY. This made sense as a safety measure when we launched. But in practice, it means the AI almost never gets to try.

Problem 2: There’s no gh pr create.

Seriously. The workflow tells Claude “push a branch with your fix.” Claude pushes the branch. And then the workflow ends. There’s zero code for creating a PR from that branch. The fix just exists as an orphaned ref that nobody will ever look at.

These five to ten missing lines of code are the entire reason the AI fix loop doesn’t work. We built the intelligence. We built the classification. We built the analysis. And then we forgot to connect the output to the thing that makes CI actually run on it.

The vision

What the loop should look like:

Main CI fails
  → investigate_failures finds blame commit
  → AI analyzes failure, writes fix, opens PR     ← currently stops here (no PR)
  → CI runs on the PR
  → If CI fails → AI reads new failure, iterates
  → Repeat until green OR max retries
  → Human reviews and merges

The key word is iterates. A fix that doesn’t compile isn’t a failure — it’s a first attempt. The whole point of having CI is that it tells you whether your code works. Right now the AI writes a fix and walks away. It should be watching what CI says about its fix and trying again.

The architecture problem: you can’t wait 90 minutes

This is where it gets interesting from a systems design perspective.

The naive approach — a single agent that writes a fix, waits for CI, reads the result, iterates — doesn’t work. CI in our monorepo takes 30 to 90 minutes. GitHub Actions has a 6-hour timeout. Claude Code has a context window. You can’t hold a stateful agent open for an hour and a half waiting for pytest to finish.

The solution is event-driven. Two workflows, loosely coupled:

Workflow 1 (exists, needs the PR creation added): AI analyzes failure, creates fix, opens PR. Exits.

Workflow 2 (new, ~100 lines): Triggers on CI completion for auto-fix PRs. If CI failed, re-runs Claude Code with the new failure context. Claude pushes to the same PR branch. CI triggers again on the push. Workflow 2 watches for that CI completion. Repeat.

Workflow 1: investigate → AI fix → create PR → exit

                    ┌──────────────────────┐
                    ↓                      │
Workflow 2: CI done on auto-fix PR?        │
              ├─ green → notify human      │
              ├─ red → AI iterate → push ──┘
              └─ max retries → give up, alert human

Max 3 attempts, then it stops and pings a human. The state is the PR itself — each push is an iteration, each CI run is feedback. No persistent agent, no long-running process, no database. Just GitHub events and workflow triggers.

The blockers, in detail

Building this revealed a pile of problems I didn’t expect. Some are plumbing. Some are design. All of them are more interesting than “the AI isn’t smart enough.”

1. The label race condition

Our investigation PRs get auto-labeled skip-code-review to bypass mandatory code review (these are bot-generated reverts, not feature code). But we have a readability lint bot that runs on every push and strips that label when it pushes its own commit.

So the sequence is: bot creates PR → adds skip-code-review label → lint bot pushes a formatting fix → label gets removed → PR is now stuck in code review limbo.

Every single revert PR in our sample showed this pattern in the GitHub label events. The workaround is to re-apply the label after the lint bot runs, but it’s a race condition that shouldn’t exist in the first place.

This matters less if we drop auto-merge and just aim for “CI green, human merges.” But it’s a symptom of a deeper problem: bots interacting with bots, creating emergent behavior nobody designed.

2. Duplicate skip PRs

The same flaky test generates a new skip PR on every CI run where it flakes. test_resume_from_yolo_checkpoint has six separate open PRs. The pipeline has duplicate detection — it checks for an existing PR with a matching title — but it’s not catching all the duplicates.

This is noise that erodes trust. If developers see six PRs for the same test, they stop paying attention to bot PRs entirely.

3. The classification gate needs data

We drew the line at “ImportError = auto-fix, CUDA error = analysis only.” But we have no data on how well this line is drawn. We don’t track: did the AI’s fix actually work? Was the analysis useful? Did the developer use it or ignore it?

Without feedback, we can’t tune the gate. We can’t answer “should TypeError be auto-fixable?” because we don’t know how the current categories are performing. The gate was designed by intuition. It should be designed by data.

4. “Revert is harder than fixing”

From the Slack data: merge commits can’t be auto-reverted. When main has advanced past the blame commit, a clean revert often introduces conflicts with later work. Developers frequently choose to fix forward instead.

This reframes the AI fix loop from “nice to have” to “sometimes the only viable path.” When reverting is impossible and the test is blocking CI for everyone, an AI that can iterate toward a working fix in minutes instead of hours is genuinely valuable. Not as a convenience — as the primary remediation strategy.

5. No feedback loop at all

If the AI’s fix is wrong, nothing happens. The branch sits there. No retry, no notification, no escalation. The workflow exits successfully because it pushed a branch, which is technically what it was asked to do.

This is the gap that Workflow 2 fills. But it’s worth noting: the system was designed without any concept of “check whether the thing you did actually worked.” It’s a fire-and-forget architecture being asked to do iterative problem-solving.

The reframe

The original ambition was auto-merge — have the bot merge its own PRs without human review. That requires:

• Code owner bypass mechanics
• Repo admin permissions
• Solving the label race condition
• Trust boundaries around AI-generated code landing without review

That’s a lot of organizational and technical surface area. And honestly? The merge button isn’t where the value is.

The descoped goal: make the AI iterate until the fix actually passes CI, then let a human click merge.

The human cost isn’t clicking “merge.” It’s the 30–90 minutes of reading logs, understanding the failure, writing a fix, pushing it, waiting for CI, finding out the fix was wrong, and doing it again. That’s what the loop automates. The merge is a two-second rubber stamp at the end.

What’s actually next

Five things, roughly in order:

1. Add gh pr create after the AI pushes a fix branch. This is literally 5–10 lines of code. It’s the single highest-leverage change in the entire system — it connects the AI’s output to CI, which is the prerequisite for everything else.

2. Build the iteration workflow. New workflow file, ~100 lines. Watches for CI completion on auto-fix PRs, re-runs the AI with the failure context, pushes to the same PR. Max 3 retries.

3. Tune the classification gate. Start logging whether fixes succeed or fail by error category. Use that data to widen or narrow the auto-fix gate. Maybe TypeError is fixable. Maybe some ImportErrors aren’t. Let the data decide.

4. Deduplicate skip PRs. The current title-matching dedup isn’t enough. Need to match on the actual test identifier, not just the PR title string.

5. Track success rate. How often does the AI’s first fix pass CI? How often does it converge within 3 attempts? What error categories have the highest fix rate? This is the data that tells us whether the system is worth investing in further.

The pattern

Here’s the thing I keep learning about CI automation: every layer you build reveals the next bottleneck.

We built failure detection. Bottleneck: nobody reads the alerts. We built bisection. Bottleneck: nobody has time to analyze what bisection found. We built AI analysis. Bottleneck: the analysis sits as a comment nobody reads. We built AI fix generation. Bottleneck: the fix sits as a branch nobody looks at.

The intelligence isn’t the hard part. Getting an AI to understand a test failure and write a plausible fix — that’s a solved problem, more or less. The hard part is the plumbing around it. Creating the PR. Triggering CI. Reading the result. Iterating. Notifying the right person. Deduplicating. Managing the label dance with other bots.

It’s not glamorous work. It’s gh pr create and workflow triggers and label race conditions. But it’s the difference between “we have AI that can fix CI” and “our CI actually gets fixed.”

The last mile is always plumbing.