CI Observability: Proving a CI Fix Actually Worked

Most CI improvements get merged with “trust me, it’s better.” Here’s what it looks like when you actually prove it.

The fix

Our container image build pipeline — Build Yolo Base and Rocket Containers, runs every 2 hours on main — had a DNS bouncing problem. When a GitHub Actions runner pushed to Azure Container Registry, the global endpoint (maifalcon.azurecr.io) could resolve to any geo-replica worldwide. If the auth token came from the US West replica but the push landed on Europe, you’d get a 401. The runner would retry, maybe hit a different replica, fail again.

The retry math was brutal: 12 retries with 180-second delays. Worst case, a single DNS hiccup could stall a job for 36 minutes. On top of that, a wait-for-image polling action sat after every push — 600-second timeout, 30-second poll interval, 30-second post-delay. The code comment on that last one said “observed to be necessary at times.” The whole action existed because a geo-replicated endpoint might resolve to a replica that hadn’t synced the manifest yet. Two methods (try az acr manifest show, fall back to docker manifest inspect), six workflow steps, all to compensate for DNS roulette.

The fix: switch from global to regional ACR endpoints. Pin the runner to one replica — maifalcon.westus3.geo.azurecr.io for Falcon, maimanifold.westus2.geo.azurecr.io for Manifold. No more bouncing. Two PRs, ten days apart:

• #19347 — Falcon ACR regional endpoints (merged Feb 15)
• #21203 — Manifold ACR regional endpoints (merged Feb 26)

Why the gap? The Falcon PR used az acr login --name maifalcon --endpoint westus3, which worked fine. Manifold needed the same thing, but az acr login --endpoint requires the service principal to have an ARM Reader role. Ours didn’t. Rather than wait for an IAM ticket, I bypassed the ARM control-plane entirely:

az acr login --name maimanifold --expose-token \
    --output tsv --query accessToken | \
  docker login maimanifold.westus2.geo.azurecr.io \
    -u 00000000-0000-0000-0000-000000000000 --password-stdin

Straight data-plane auth. The 00000000... username is the convention for token-based ACR login — it tells the registry “this isn’t a user, it’s a bearer token.” No ARM role needed.

Along with the endpoint swap, both PRs also gutted the retry and polling config:

After merging, the obvious question: did it actually help?

What to measure

Two things should improve:

1. Duration — less retry wait, no polling loops. Before: up to 36 min worst case. After: 3 retries x 30s = 1.5 min worst case, no polling.
2. Failure rate — DNS bouncing caused intermittent blob unknown errors and 401s. With a pinned replica, those should drop to zero.

But you can’t just look at the whole pipeline. Most jobs in the build workflow weren’t affected — the actual image build (the buildx bake step) takes the same time regardless. Measuring avg(pipeline duration) would drown the signal in noise from the build and test jobs.

Mapping changes to jobs

Each PR touched specific steps in the GitHub Actions workflow. The affected jobs:

This is the part people skip. If you don’t know exactly which observable components changed, you’re measuring vibes.

The Datadog query

We use Datadog CI Visibility, which indexes GitHub Actions job-level data. The query that filters to exactly the affected jobs:

ci_level:job
@ci.pipeline.name:"Build Yolo Base and Rocket Containers"
@ci.job.name:(
  "Upload image to Manifold ACR"
  OR "Merge and push multi-arch image to Manifold ACR"
  OR "Merge and push yolo multi-arch image to Falcon ACR"
)

Settings that matter:

• Measure: avg(@duration) for the duration chart, count for failure rate
• Visualization: Timeseries — you want to see the step function at merge time
• Time range: 4 weeks, to capture both PR merge dates (Feb 15 and Feb 26)
• Group by: @ci.job.name so each job gets its own line
• Don’t filter to @ci.status:success — you want to see failure rate changes too

The 4-week window is important. Too short and you don’t have enough “before” data. Too long and you’re looking at noise from unrelated changes.

The bimodal distribution

Before I merged the Manifold PR, my teammate Masatoshi shared a distribution chart of the Manifold job durations. It was multimodal — clusters at different duration bands, with a long tail stretching past 12 minutes.

The clusters almost certainly corresponded to which ACR replica the runner happened to hit. A nearby replica: fast. A far replica: slow, plus retries. The between-PRs data confirmed this — Manifold durations ranged from 45 seconds to 605 seconds, a 13x spread for the same operation. That’s not variance, that’s a slot machine.

With regional endpoints pinning every request to one replica, those clusters should collapse into one — right around where the fastest cluster was. That distribution chart was better evidence than any single metric. It showed the mechanism of the problem, not just its symptoms.

What the data showed

Falcon (PR #19347, merged Feb 15)

The Datadog CI Visibility data shows blob unknown push failures on Feb 13 — the last two before the fix landed:

Each failure burned 14-23 minutes of compute time — the build runs to completion, then the push fails on a replica that doesn’t recognize the blob, and retries chew through the rest.

After the merge on Feb 15: zero blob unknown errors. Two transitional failures on merge day itself (old code still in the queue, still pushing to the global endpoint), then clean. Here’s the Datadog view — the cluster of failures before Feb 15, then nothing.

(Datadog CI retains 30 days of events, so earlier blob unknown failures have rotated out. The pattern was ongoing — these weren’t one-offs.)

Worth noting: the Falcon duration didn’t visibly improve. The multi-arch merge job is still spiky — that’s normal variance from imagetools create depending on layer sizes and registry load. The win here is purely reliability: the job stopped failing, not that it got faster.

The gap (Feb 15-25): Manifold still broken

Falcon was fixed, but Manifold was still on global endpoints. During these 10 days, the Manifold multi-arch merge job hit 6 auth failures (Feb 17-18): all Azure CLI login failed after 3 attempts or OIDC errors. Same DNS bouncing, different registry.

This was actually useful as a natural control group. Same pipeline, same runners, same time period — but one registry pinned, one still bouncing.

Manifold (PR #21203, merged Feb 26)

Post-fix Manifold multi-arch merge: 35-87 seconds, median ~50s. Compare that to the 45-605 second range during the gap period — a 13x spread collapsed to 2.5x. The slot machine became a vending machine.

Zero Manifold errors since merge. The duration timeseries shows the variance collapse at the merge date.

Using Claude Code to build the dashboard

Here’s where it got interesting. I didn’t build those Datadog queries by hand.

I asked Claude Code which jobs were affected by each PR. It read the workflow YAML, traced the changes, and listed the job names. Then it constructed the Datadog query — with proper URL encoding, pre-configured visualization settings, correct measure and grouping — and handed me a direct link.

One prompt to go from “I just merged a PR” to “here’s a dashboard showing the impact.”

This is a useful pattern: AI as the bridge between “what changed in code” and “where to look in monitoring.” The model knows the codebase (which jobs map to which workflow steps) and knows the monitoring tool’s query syntax (Datadog’s faceted search). It connects the two faster than you can click through menus.

I’m not saying this is the killer app of AI in CI. But it’s a real quality-of-life thing. The friction of constructing monitoring queries is just high enough that most people don’t bother — and then the improvement goes unproven.

The general pattern

If you’re making CI improvements and want to prove they worked:

1. Before the change — decide what metrics should improve. Duration? Failure rate? Flake rate? Be specific.
2. Map code to observables — which CI jobs, steps, or pipelines will be affected? Not the whole pipeline — the specific components.
3. Construct the query — filter to exactly those components. Your monitoring tool probably supports job-level filtering; use it.
4. Set the time range — capture enough before-and-after data. A day isn’t enough. 2-4 weeks is usually right.
5. Share the evidence — a Datadog link is worth more than “it works on my machine.” Drop it in the PR, drop it in Slack. Now it’s documented.

None of this is revolutionary. It’s just the discipline of closing the loop — treating CI work with the same rigor you’d apply to a production deployment. The fix isn’t done when the PR merges. It’s done when the dashboard shows the improvement.