Wiring AI Agents into CI: From Cold DM to Working Pipeline

I wanted Claude to automatically diagnose CI test failures. Two days later, it’s running in production — but the path there looked nothing like what I planned.

The idea

Our CI pipeline already does git bisection when a test breaks. It finds the blame commit, creates a skip or revert PR, and posts to Slack. Solid, but dumb — it tells you what broke, not why.

The idea: after bisection, hand the error text and blame diff to Claude. Get back a root cause summary, a suggested fix, and a confidence level. Stick it in the PR description and Slack message. Engineers land on the thread and immediately know what happened instead of digging through logs.

I wrote the whole thing — a Python module calling the Anthropic API, wired into three decision points in the investigation pipeline. Even had tests with mocked API calls. Draft PR up, feeling good.

One problem: I needed an ANTHROPIC_API_KEY as a repo secret.

The cold DM

I found Cam Immesoete on Slack — he’s on the Picasso team, active in the AI agents and DevEx space. Cold messaged him asking about API key provisioning.

His reply flipped my entire approach:

We in fact do not interface with the Anthropic API in any way. We take a different approach. We interface with Claude via the Google Vertex AI framework, and have a GCP account for this.

Turns out our org already has Vertex AI set up with org-wide secrets provisioned. No API key needed. No new secrets to create. The infrastructure was already there — I just didn’t know about it.

Cam offered two things: Reader access to their mai-agents repo as a reference implementation, and to install their claude-code GitHub App on our repo. The whole setup took maybe 30 minutes of Slack back-and-forth — he JIT’d into yolo, installed the app, done.

The reference implementation

The key file is shared-claude-code-review.yml in the mai-agents repo. It’s a reusable workflow for AI code review on PRs, but the auth and infra patterns are what I needed.

Here’s what their setup looks like:

# Auth: GCP Workload Identity Federation (no stored API keys)
- uses: google-github-actions/auth@v2
  with:
    workload_identity_provider: ${{ vars.COCO_GCP_WORKLOAD_IDENTITY_PROVIDER }}
    service_account: ${{ vars.COCO_GCP_SERVICE_ACCOUNT }}

# Claude Code: Vertex AI mode
- uses: anthropics/claude-code-action@v1
  with:
    use_vertex: "true"
    github_token: ${{ steps.app-token.outputs.token }}
    claude_args: |
      --model claude-sonnet-4-5@20250929
      --max-turns 100
  env:
    ANTHROPIC_VERTEX_PROJECT_ID: coco-mai
    CLOUD_ML_REGION: global
    CLAUDE_CODE_USE_VERTEX: "1"

A few things stood out:

• No API keys anywhere. Workload Identity Federation means the GitHub Actions runner authenticates to GCP via OIDC tokens. No secrets to rotate.
• Org-wide variables. CLAUDE_CODE_APP_ID, COCO_GCP_WORKLOAD_IDENTITY_PROVIDER, COCO_GCP_SERVICE_ACCOUNT — all stored at the org level. Any repo that installs the app gets them for free.
• Retry logic. They found that 52.6% of their workflow failures were transient GCP OIDC issues. So they run Claude Code with continue-on-error: true, wait 30 seconds, and retry once. Simple and effective.
• Vertex model naming. It’s claude-sonnet-4-5@20250929, not claude-3-5-sonnet-20241022. The @ version format is a Vertex thing.

Their workflow also has a plugin system — it checks out the mai-agents repo, copies agent/skill markdown files into ~/.claude/, and cleans up. Extensible review behavior without touching the core workflow. Pretty slick.

Adapting it

My use case is different — failure investigation, not PR review — but the infrastructure layer is identical. Here’s what I changed in my workflow:

The prompt stayed the same — Claude reads the failing test, reads the blame diff, identifies root cause, and optionally writes a fix on a new branch. The only change was how it authenticates to Claude.

The final workflow: ai_fix_failure.yml — triggered via workflow_dispatch with test ID, error text, blame SHA, and package name. GCP auth, Claude Code with Vertex AI, retry on failure.

Testing it

Triggered the workflow from CLI with a dummy test failure:

gh workflow run ai_fix_failure.yml \
  --ref shiyuanzheng/claude-code-action-fix \
  -f test_id="ci/tests/tools/test_investigate.py::test_post_to_slack" \
  -f error_text="AssertionError: expected call not found" \
  -f package="ci"

Watched the steps light up green one by one:

First attempt passed. The GCP auth worked, Claude initialized with the right model, read the test file, analyzed the error, and completed. No retry needed.

The whole thing — from Cam installing the app to a successful run — took about two hours.

What I threw away

My original Approach A — the Python SDK calling Anthropic directly — is now parked. It works, it has tests, but it’s solving the wrong problem. The org doesn’t use the Anthropic API directly. Fighting that would mean maintaining a separate auth path, provisioning a separate API key, and being the only repo doing it differently.

Sometimes the best engineering decision is throwing away working code when the org has a better pattern.

What’s next

Right now the workflow is manual — someone has to trigger it with test details. The next step is wiring it into the existing investigation pipeline so it fires automatically when bisection finds a blame commit.

The pipeline would look like:

Test fails in CI
  └── investigate_failures.yml (bisects, finds blame)
        └── triggers ai_fix_failure.yml
              ├── Claude reads test + blame diff
              ├── Identifies root cause
              ├── Writes fix on a new branch
              └── (future: creates a fix PR)

Currently it commits to a branch but doesn’t create a PR — intentionally conservative. Once we trust it, we’ll let it open PRs.

Other things on the list:

• Wire in the mai-agents plugin system — their review agents and skills could enhance failure analysis
• Structured output — parse Claude’s response into root cause / fix / confidence fields for Slack formatting
• Metrics — track how often the AI diagnosis is accurate, how often its fixes are accepted

The meta-lesson

I spent a day building a thing. Then a 5-minute Slack conversation redirected the entire approach and made it better. The code I threw away was fine — the infrastructure decision was wrong.

If you’re at a big org and about to provision a new API key or spin up a new service, DM someone first. The thing you need might already exist three teams over.