The AI CI Helper’s First Real Test (It Failed Its Own Test)

Teammate pinged me in #ci-cd-alerts. GPU test failure on main — test_enable_dp_lm_head in sgl_server, timed out at 300 seconds. Datadog monitor fired, Slack alert landed, and the question was simple: “can AI fix run this?”

The answer turned out to be more interesting than yes or no. And then the same test came back two weeks later for a rematch.

The pipeline fires, finds nothing

The automated investigation workflow did trigger. Run 22971605267, kicked off about 3 minutes after the CI failure. All jobs completed successfully. Green checkmarks across the board.

But it found zero test failures. Zero AI analysis requests:

Searching for failures in ./artifacts matching pytest-*.json
No failures found.

A weird thing for a failure investigation pipeline to say about a failure that definitely happened.

Tracing the artifact mismatch

Pulled the workflow logs and test artifacts. Here’s what I found.

The test was detected. The pytest JSON report had outcome: "failed" at the top level — pipeline correctly identified it as a failure. But the error text it extracted was empty. No traceback, no timeout message, nothing.

A failure with no content is invisible to everything downstream.

The bug: GPU and CPU pytest JSON reports structure their error details differently.

CPU format:

{
  "outcome": "failed",
  "longrepr": "AssertionError: expected 42, got 41\n..."
}

GPU format:

{
  "outcome": "failed",
  "call": {
    "longrepr": "TIMEOUT: test exceeded 300s limit\n...",
    "crash": {
      "message": "TimeoutError"
    }
  }
}

find_failures_in_artifacts() was looking for test["longrepr"]. GPU tests put it at test["call"]["longrepr"]. Function found the failure, extracted nothing, moved on.

One-line fix. Cascade the lookup: test.get("longrepr") or test.get("call", {}).get("longrepr") or test.get("call", {}).get("crash", {}).get("message", ""). But the finding took longer than the fixing. It always does.

Manual trigger, real results

Since the automated pipeline missed the error context, I triggered ai_fix_failure.yml manually. Downloaded the test artifact, extracted the error text, passed it in with the test ID and Slack thread info.

Claude Sonnet 4.5 via Vertex AI came back in about 3 minutes with a structured analysis. Classification: ANALYSIS ONLY — correct call, because a timeout on a GPU test isn’t something you can auto-fix without actually running on GPUs.

Four real issues found:

1. The test is an incomplete smoke test. Creates Engine(server_args) but never asserts anything. The entire test is “start the engine and see if it crashes.” That’s not a test, that’s a prayer.

2. Missing engine.shutdown() cleanup. Resources leak if the test fails or times out, poisoning subsequent tests in the same session.

3. Resource requirements are brutal. --tp=4 --dp=2 means 8 GPU processes. If the CI runner doesn’t have 8 GPUs available, the test hangs until timeout kills it. Which is exactly what happened.

4. Cross-reference to DEVEX-360. A similar test in the same package was already skipped for identical flakiness. The AI found this by searching the codebase for pytest.mark.skip annotations mentioning related test names.

That last one is the interesting find. Cross-referencing a failure against existing skip annotations requires knowing the codebase well enough to connect dots. A human triaging this would probably find issues 1-3 in 15-20 minutes. Issue 4 depends on whether they remember DEVEX-360 exists — and nobody remembers ticket numbers six weeks later.

Analysis got posted back to the Slack thread automatically. Zero false positives. Three minutes wall clock.

The brainstorm session

My teammate Yipu saw the analysis in Slack and immediately started riffing on extensions. Quick back-and-forth right in the thread — the kind of conversation that only happens when you have a working demo in front of you.

Four changes total — one bug fix, three features — all implemented in the same session and landed in PR #24660:

Feature A is the most impactful. Our investigation pipeline classifies failures as flaky, baseline-broken, or regression. Only regressions (the bisection path) dispatched AI analysis. Flaky tests got a @pytest.mark.skip PR and a Linear ticket — no root cause analysis at all.

But flaky tests are often the most interesting to analyze. “This test passes sometimes and fails sometimes” is a richer debugging problem than “this test broke at commit X.” The AI can look at the test code and spot non-determinism sources — timing dependencies, shared state, resource contention, missing mocks. Exactly the kind of analysis that saves human time.

The deeper conversation

That Slack thread was the demo. The real brainstorm happened in DMs a couple weeks later when the same test came back.

Yipu asked me straight up: “Is it true that finding a blame commit through bisection is actually rare? I haven’t seen an auto-fix land yet.”

Fair point. Bisection works when there’s a clean regression — commit A passes, commit B fails. But a lot of CI failures aren’t clean regressions. They’re flaky tests that finally got unlucky, infra issues, resource contention, timing-dependent garbage.

So we started mapping out where AI analysis could actually help beyond bisection:

• Flaky tests — analyze the test code, figure out why it’s flaky, not just that it’s flaky. Timing dependency? Shared state? Missing teardown? The AI can read the test and tell you.
• Infra errors — build failures, registry timeouts, runner issues. AI probably can’t fix these, but it can suggest retry strategies or at least classify them so on-call doesn’t waste time investigating.
• Fixable patterns — some flaky tests have mechanical fixes. Missing cleanup, hardcoded timeouts, race conditions with known solutions. Worth trying an auto-fix PR for those.

But here’s where Yipu said something that stuck. She compared our current Datadog test retry approach to 掩耳盗铃 — literally “plugging your ears while stealing a bell.” The Chinese idiom for self-deception. DD retry detects flaky tests, retries them, marks the workflow as passing, and logs it in Datadog. Problem solved, right?

Not really. The workflow passes. The test is still flaky. The customer-facing test suite still has a test that fails randomly. DD retry helps on-call by reducing alert noise — which is genuinely valuable — but it accepts that flaky tests exist without doing anything about the root cause. It’s noise reduction, not signal improvement.

That framing changed how I think about this whole system. Retry is layer one: stop the bleeding. Investigation and AI analysis are layer two: actually diagnose and fix the wound.

Same test, different day

March 11. Two weeks after the original incident. The same test_enable_dp_lm_head timed out again on main.

But this time, everything was different.

First, Jay’s new Datadog Slack integration fired — its very first real alert. Yipu spotted it immediately: “this is the first one” with a smiley. The DD beta caught the failure with structured details and direct links to the test run. Our in-house bot wouldn’t have flagged this one at all — it was an infra/build failure pattern that fell outside the old bot’s detection scope.

Then the full pipeline kicked in. Investigation bot started about 12 minutes after the alert, ran the test four times, watched it pass all four, classified it as flaky. Auto-created skip PR #24586 and Linear ticket DEVEX-543. Standard procedure.

And then — the part that didn’t exist two weeks ago — AI analysis ran and posted structured findings to the thread. Classified as ANALYSIS ONLY (correct). Identified the root cause: multiprocessing/subprocess communication failure during Engine initialization. The test legitimately needs more than 300 seconds to spin up 8 GPU processes, and the timeout was too aggressive.

The bug fix from the original incident was deployed and working. GPU test artifacts were being parsed correctly. The full pipeline — detect, investigate, classify, analyze — ran end-to-end without manual intervention.

The test owner Praveen showed up in the thread and confirmed: yes, the test legitimately needs >300 seconds. Ryan suggested @pytest.mark.timeout(500) and marking it slow. Yipu suggested moving it to daily SGL tests entirely if it needs more than 5 minutes. A real conversation about the right fix, informed by structured analysis, happening in the same thread where the alert landed.

That’s the workflow actually working. Not “we built a thing and it should work.” It worked. On the same test that exposed the original bug.

The alert exhaustion problem

Here’s the context that makes all of this urgent.

I pulled two weeks of data from #ci-cd-alerts. The numbers are brutal:

Twenty-nine alerts on a Tuesday. That’s an alert every 30 minutes during work hours. Nearly 100% bot-originated. The top threads — all “Main Required CI Failed on Main Branch” — had 25 to 46 replies each.

One person replied in 47 threads in a single week. That’s not on-call rotation. That’s a full-time job that nobody signed up for.

And the thing is — most of these alerts are flaky tests. Tests that pass on retry but fail often enough to fire the bot. Each one generates a Slack thread, gets investigated, gets discussed, maybe gets a skip PR. Multiply that by 12 a day and you’ve got a team spending more time managing test flakiness than writing code.

This is why “just retry it” isn’t enough. Retry reduces the alert count. It doesn’t reduce the flakiness. The tests are still broken. The on-call person is still drowning.

Three layers

The squad’s been converging on a three-layer approach. Not from a design doc — from watching what actually works and what doesn’t.

Layer 1: Noise filter (DD auto-retry)

Jay proposed enabling Datadog’s automatic test retry on main CI. When a test fails, DD retries it. If it passes on retry, the workflow stays green and DD logs it as flaky. No alert, no Slack thread, no investigation.

This is the 掩耳盗铃 layer — and that’s fine, as long as you know it’s just layer one. The immediate win is massive: Ryan did the math on current flaky test handling. Detection takes ~90 minutes. The disable PR takes an hour. CI pickup takes another 45 minutes. That’s 4 hours from “test is flaky” to “test stops bothering people.” DD quarantine from the UI could cut that to minutes — no PR needed, no developer approval.

The squad agreed: drop Phase 1 of the investigation pipeline (the “is it flaky?” check) since DD retry handles that natively now. Keep Phase 2 (baseline check) and Phase 3 (bisection) — those catch real regressions that retry can’t fix.

Layer 2: Investigation pipeline

The existing pipeline. Detects failures, reruns tests, classifies them as flaky/baseline-broken/regression, runs bisection on regressions, auto-creates skip PRs for flaky tests. This is the layer that actually does something about failures.

With DD retry handling the noise, the investigation pipeline can focus on what it’s good at: the failures that aren’t flaky. Real regressions. Tests that started failing at a specific commit. The cases where bisection actually finds a blame commit and an auto-fix branch has a shot at working.

Layer 3: AI analysis

Root cause analysis for everything the pipeline touches. Flaky tests get analyzed for why they’re flaky — not just skipped. Regressions get analyzed for fix strategies. Infra failures get classified and routed.

This is the layer that turns “we know this is broken” into “we know why this is broken and here’s what to do about it.”

Yipu’s key push: alerts should be immediate, not daily summaries. If a test gets retried by two separate workflow runs, message #ci-cd-alerts right then. Daily digests are how things fall through cracks.

And there’s a dashboard now — built on DDSQL — that computes per-test failure rates on main versus all branches, using DD’s built-in is_flaky detection. Same commit, both pass and fail, equals flaky. Columns for failure count, failure rate, latest failure timestamp, total runs. The data exists. It’s visible. Now it needs to drive action, not just observation.

Numbers

The 3-minute AI analysis versus 30-minute human triage is the headline number, but it’s not the real value. The real value is the brainstorm that happened because the analysis was fast.

When you get a structured root cause analysis in 3 minutes instead of 30, your brain doesn’t context-switch away. You’re still in the problem. You see the analysis and immediately think “wait, we should also handle flaky tests” and “what about infra failures?” That’s how four changes came out of one incident.

What this validates (and what it doesn’t)

Validates:

• AI analysis quality is genuinely useful. Senior-engineer-level triage in 3 minutes, with cross-references a human might miss.
• The pipeline architecture (investigate -> classify -> dispatch) is fundamentally sound.
• Real incidents drive better feature ideas than roadmap planning sessions.
• The bug fix works. Two weeks later, same test, full pipeline ran correctly without manual intervention. GPU artifact parsing is solid.
• The three-layer model holds up. DD retry for noise, investigation for real breaks, AI for root cause — each layer does something the others can’t.

Doesn’t validate:

• Auto-fix capability. Both incidents were ANALYSIS ONLY. We still haven’t seen a real-world auto-fix land from this pipeline.
• Scale. Two incidents involving the same test is a data point, not a dataset. We need the evaluation pipeline to know if this quality holds across failure types.
• Whether AI analysis on flaky tests actually leads to fixes. The theory is sound — analyze why it’s flaky, suggest a fix, maybe even auto-PR it. But we haven’t closed that loop yet.
• Alert exhaustion reduction. The numbers say ~12 alerts/day. DD retry should cut that significantly. But “should” isn’t “did.”

The auto-fix gap is the big one. Analysis is valuable — it saves triage time, it surfaces things humans miss, it keeps context in the thread where the alert landed. But the dream is a pipeline that doesn’t just tell you what’s wrong, it fixes it. We’re not there yet. The brainstorm with Yipu mapped out the path — flaky test analysis, mechanical fix patterns, auto-PR for known solutions — but that’s roadmap, not reality.

What is real: the same test that broke our pipeline came back, and this time the pipeline handled it. That’s the smallest possible proof that the system works. Now we need to see it work on a hundred different failures, not the same one twice.

The lesson about schema assumptions

If there’s a generalizable takeaway: when you’re building a pipeline that processes test artifacts, don’t assume the artifact format is uniform across test environments.

CPU tests run with standard pytest. GPU tests run with a distributed test harness that wraps pytest and adds its own reporting layer. The outer schema is the same — JSON file with test results — but the inner structure of each test entry differs. The outcome field is always at the top level (thank god), but error details migrate depending on how the test runner captures them.

The fix is defensive extraction with fallback paths. But the real fix is testing your artifact parsing against examples from every test environment, not just the one you had handy when you wrote the function.

We had CPU test fixtures in our unit tests. We did not have GPU test fixtures. Because of course we didn’t — who writes unit tests for artifact parsing that include fixtures from a GPU cluster?

The person who’s been burned by this, that’s who. So now we do.