Two teams in the same monorepo, same quarter, independently wired Claude into their CI failure pipelines. Same action (anthropics/claude-code-action@v1), same model (claude-sonnet-4-5@20250929 on Vertex AI), same goal — “tell us why CI broke so we don’t have to read logs.” They arrived at very different systems.
I built one of them. Here’s what I learned from comparing them side by side.
The setup
System A is the Rocket team’s analyze_rocket_failure. They run SLURM-scheduled reinforcement learning jobs on a Condor cluster — off-policy training, eval, SFT, SGL generation. Four times a day, cron kicks off regression tests. When they fail, the pipeline SSH’s into the cluster, collects experiment logs (stderr files, RayJobs output, per-actor detailed logs), uploads them as GitHub Actions artifacts, and hands them to Claude.
System B is ours — the DevEx squad’s investigate_failures.yml. It triggers on workflow_run whenever Main Required CI fails. We’re dealing with CPU/GPU test shards across a hundred-package Python monorepo. Failures could be anything: an import renamed in package A breaking package B, a CUDA OOM, a flaky race condition, a transient ACR DNS blip.
Both systems end the same way: a threaded Slack reply under the failure notification with Claude’s analysis. But how they get there is where it gets interesting.
The divergence
The fundamental question both teams answered differently: how much work should happen before Claude sees anything?
Rocket’s answer: not much. Collect the raw logs, pass them to Claude, let it figure it out. Claude gets read-only tools — grep, find, head, tail, cat, ls — and a detailed prompt about SLURM/Ray/NCCL log patterns. It reads the logs, classifies the failure (transient vs persistent, NCCL timeout vs OOM vs configuration error), and writes a Slack-formatted summary to a file. A separate step reads that file and posts it.
Our answer: a lot. Before Claude sees anything, a Python CLI (ci.cli investigate-failures) runs git bisection, identifies blame commits, categorizes failures (blame vs flaky vs infra), creates revert or skip PRs, files Linear tickets, and posts initial Slack notifications. Then it writes structured JSON files — test ID, package name, blame SHA, error text, failure type — to an ai-analysis-requests/ directory. Claude picks those up with git tools (git diff, git log, git show), reads the blame diff, reads the test code, and either attempts an auto-fix or writes a structured analysis.
Here’s the comparison that matters:
| Aspect | Rocket (System A) | DevEx (System B) |
|---|---|---|
| Pre-processing | Minimal — raw logs | Heavy — bisection, blame, categorization |
| Claude’s role | Log analyst | Code analyst + potential fixer |
| Claude’s tools | Read-only | Read + Edit + git |
| Can Claude fix things? | No | Yes (for simple errors) |
| Max turns | 30 | 50 |
| Output | Slack summary file | Structured analysis + fix branches |
| Git context | None | Full — blame commits provided |
Why the divergence makes sense
Rocket’s failure modes are predictable. SLURM jobs fail in recognizable ways — NCCL timeouts, job requeueing, OOM kills, Ray actor crashes. The log format is consistent. The error signatures are finite. A good prompt can enumerate them.
Their prompt literally tells Claude: “Look for CANCELLED DUE TO JOB REQUEUE — that means the SLURM scheduler preempted the job, it’s transient.” And: “NCCL errors fall into three buckets: timeout (usually network), initialization failure (usually config), and hardware error (check GPU health).” When your failure taxonomy is small and stable, you can encode it in the prompt and let Claude match patterns.
Our failure modes are not predictable. Any of 100+ Python packages could break for any reason. An ImportError in mai_layers looks nothing like a CUDA error in rocket_lib looks nothing like a TimeoutError in a flaky integration test. The error could be caused by a commit from three hours ago that changed an unrelated package’s dependency graph. You can’t enumerate all the patterns in a prompt — there are too many.
So we do the narrowing before Claude. By the time Claude sees a failure, it already knows: “Test test_training in package mai_layers started failing after commit abc123 by Alice. Here’s the diff. Here’s the error.” Claude’s job isn’t pattern-matching against a log format — it’s reading code and understanding why a specific change broke a specific test.
The tradeoff is complexity. Our system is hundreds of lines of Python. Theirs is mostly a prompt.
What we should steal from each other
From Rocket: the file-output pattern. Their Claude writes to ./claude_analysis_summary.txt. A separate step reads it and builds the Slack payload. Clean separation. Our system extracts Claude’s analysis from the execution_file JSON, which is more fragile — you’re parsing the action’s internal output format. Writing to an explicit file is simpler and more testable.
From Rocket: domain-specific log patterns in the prompt. We run GPU tests too. Their NCCL error classification section — timeout vs init failure vs hardware error — is directly reusable. Right now if one of our GPU tests hits an NCCL failure, Claude doesn’t have that taxonomy. It should.
From Rocket: tool restriction. They give Claude read-only access. For log analysis, this is correct — you don’t want Claude editing experiment logs. We give Claude Edit and Bash(git *) because it attempts auto-fixes, which is a valid tradeoff for our use case. But it’s worth being explicit about: more tools = more ways for Claude to do something unexpected.
From us: preprocessing narrows the search space. Rocket could benefit from even light preprocessing — extracting the failing job name, the exit code, the last N lines of stderr before the crash. Giving Claude 500 lines of log where the error is on line 487 is making it do work that tail already solves.
From us: the classification gate. Our prompt explicitly tells Claude what to auto-fix (ImportError, AttributeError, NameError, SyntaxError) and what to only analyze (CUDA errors, OOM, NCCL, timeouts, complex logic). Without this gate, Claude will try to fix CUDA out-of-memory errors by… editing the test to use less memory. Which is technically a fix. It’s just not the right one.
From us: automated follow-up. We create Linear tickets for flaky tests, revert PRs for blame failures, skip PRs for persistent flakes. Their system is notification-only — the team discusses in Slack and acts manually. The Slack thread is ephemeral. The Linear ticket persists.
Design details that matter
The threaded reply model. Both systems do this: the failure notification is the parent message, Claude’s analysis is a thread reply. This is the right UX. The engineer sees the red dot, opens the thread, gets context. Neither system tries to put the AI analysis in the notification itself — it’s supplementary, not primary. Good instinct from both teams independently.
Output format constraints. Rocket enforces Slack mrkdwn (no # headers, bold with *asterisks*, 3000 character limit). Our system writes structured markdown that gets reformatted for each destination (PR comment vs Slack). Rocket’s approach is simpler. Ours is more flexible. Depends on how many output surfaces you have.
Retry strategy. Rocket retries once immediately. We retry once with a 30-second backoff. Both learned from the mai-agents team’s discovery that ~52% of Claude Code Action failures are transient GCP OIDC issues. The backoff helps, but honestly, one retry catches most of it either way.
Claude writes a file, not stdout. This one is worth repeating. Both systems have Claude write its output to a file rather than relying on parsing the action’s execution output. Rocket does claude_analysis_summary.txt. We write structured JSON. If you’re building a Claude Code Action workflow, this is the pattern — give Claude a known output path, read it in a subsequent step. It’s more reliable than parsing execution logs.
The meta-lesson
Both teams independently arrived at the same architecture: Claude Code Action → Vertex AI → Slack notification. The infra layer is identical because the org converged on it (GCP Workload Identity, org-wide variables, the claude-code GitHub App). The divergence is entirely in the application layer — specifically, in how much intelligence lives in the prompt vs in the code.
There’s a spectrum here:
All prompt ←————————————————————————→ All code
(Rocket) (DevEx)
The right position depends on one thing: how predictable are your failure modes?
If your failures follow a small set of known patterns — same log format, same error categories, same recovery actions — lean toward prompt. It’s faster to build, easier to iterate on, and Claude is genuinely good at pattern matching against well-described schemas.
If your failures are wildly varied — any package, any error type, any root cause, any commit in the last week — lean toward code. Preprocessing narrows the search space before Claude even starts. Git bisection, blame identification, and failure categorization are all deterministic algorithms that shouldn’t burn LLM tokens.
Most teams should probably start where Rocket started — minimal preprocessing, good prompt, read-only tools — and move toward our approach only when they find Claude floundering on the variety. You’ll know it’s time to add preprocessing when Claude’s analyses start saying “the error appears to be related to a recent change” without being able to tell you which change.
The ironic part: Rocket’s system is currently broken (a GitHub App Token step is failing), and ours is running in production. The simpler system went down first. Make of that what you will.