Docker Builds: Debugging Stuck Builds with Datadog and the GH API

2026/03/05

BUILDcidebuggingdockerinfra

A PR build got stuck. The docker buildx bake step on the amd64 Falcon ACR job just… stopped producing output. Fifty-five minutes of silence, then the job timeout killed it. Classic intermittent CI failure — the kind where you stare at the logs and there are no logs to stare at.

Turns out the job wasn’t flaky. The runner was sick. But it took cross-referencing two completely different data sources to see that.

The symptom

I noticed it on a PR run. The “Build and push yolo image to Falcon ACR (amd64)” job was stuck on the Build image step. No log output. No progress. Just a spinner.

The arm64 job on the same pipeline run? Finished in 16 minutes. Same cache config, same registry, same workflow file. The only difference was the runner architecture — and apparently, the runner’s will to live.

Four attempts, three different failure modes

This run went through four attempts, each failing differently:

Attempt 2 was actually making progress — 32 cached Docker layers, build humming along — then SIGTERM. Dead. The workflow has cancel-in-progress: true in its concurrency config, so when someone re-ran the workflow, it killed the previous attempt mid-build. Your build was fine. Someone else’s click murdered it.

Attempt 3 — the one I originally noticed — hung for 55 minutes on Build image with zero output. The 60-minute job timeout eventually mercy-killed it. No logs captured because the step was still in_progress when it died. You can’t diagnose a ghost.

Attempt 4 got the same hang, then got cancelled when a new commit pushed to the PR.

Three failures, two different root causes (cancel-in-progress vs. build hang), all looking identical in the GitHub Actions UI: red X, failed job, no useful output.

The wrong rabbit hole

My first instinct was to check if this was a broader pattern. Were the scheduled main branch builds also failing? I checked the two most recent failures on main — runs 22602259935 and 22606311804.

They were failing. But on a completely different job: Build rocket_yolo for Falcon (amd64), not the yolo pretraining image. The yolo image job had succeeded fine on both of those runs.

Same pipeline, different job, different failure. If I’d just glanced at the pipeline-level status and assumed it was the same issue, I’d have been debugging the wrong thing.

Two data sources, one picture

Here’s where it got interesting. The GitHub Actions API gives you the per-run story — step-by-step logs, attempt history, exact timestamps. But it can’t answer aggregate questions. Is this job normally flaky? How often does it fail? Is the failure correlated with anything?

Datadog CI Visibility gives you the aggregate story — failure rates, duration distributions, trends over time. But it can’t tell you what happened inside a specific run. And — important detail — PR-triggered builds don’t show up in our Datadog CI data at all. That’s a monitoring blind spot we didn’t know about until now.

So I used both.

What GitHub told me

The gh CLI is underrated for this kind of investigation. You can pull job details, step timings, and attempt history without clicking through the UI:

# Get all jobs for a specific run attempt
gh api repos/infinity-microsoft/yolo/actions/runs/22599075016/attempts/3/jobs

# Get step-by-step timing for a specific job
gh api repos/infinity-microsoft/yolo/actions/runs/22599075016/jobs/65489455420

From the attempt history, I could see the pattern: attempt 2 was killed by cancel-in-progress, attempts 3 and 4 genuinely hung. Different failure modes masquerading as the same thing.

What Datadog told me

For the aggregate picture, I queried Datadog’s CI pipeline events API:

curl -s "https://${DD_SITE}/api/v2/ci/pipelines/events/search" \
  -H "DD-API-KEY: ${DD_API_KEY}" \
  -H "DD-APPLICATION-KEY: ${DD_APP_KEY}" \
  -H "Content-Type: application/json" \
  -d '{
    "filter": {
      "from": "now-14d", "to": "now",
      "query": "ci_level:job @ci.pipeline.name:\"Build Yolo Base and Rocket Containers\" @ci.job.name:\"Build and push yolo image to Falcon ACR (amd64, ubuntu-2404-x64-32c.128g.1200g)\""
    },
    "page": {"limit": 50},
    "sort": "-timestamp"
  }'

The yolo amd64 job on main? 50 events in 14 days. All success. Average duration 12.8 minutes. P50 at 751 seconds, P90 at 925 seconds. Rock solid. Whatever was killing my PR build wasn’t happening on main.

But rocket_yolo was a different story. 12 errors in 14 days — 19% failure rate. Every single error had the same signature: buildx bake hangs until the 90-minute timeout kills it. Normal runs took ~22 minutes. Failed runs showed durations of 53-58 minutes — the exact timeout ceiling, not the actual build time.

The smoking gun: runner IDs

This is the part you can only see in Datadog, because it captures the runner node name as a field.

When both the yolo image job AND the rocket_yolo job failed in the same pipeline run, their runner IDs were sequential — like runner-abc-88720 and runner-abc-88721. Sequential IDs mean both jobs landed on the same physical host. When that host was sick — disk I/O issues, network problems, whatever — everything scheduled on it hung.

When the yolo job succeeded but rocket_yolo failed? Different runner hosts. The yolo job got a healthy node; rocket_yolo got the bad one.

This wasn’t a Docker build problem. It wasn’t a cache problem. It wasn’t an ACR connectivity problem. It was a runner infrastructure problem — specific hosts intermittently becoming unresponsive, causing buildx bake to hang on I/O operations until the timeout killed it.

You can’t see this from a single run’s logs. You need aggregate data with runner-level correlation. One data source gives you the symptoms; both together give you the diagnosis.

The timeout trap

Here’s the thing that made this worse than it needed to be. Both jobs use wretry with attempt_limit: 3 for automatic retries. Great idea in theory — if buildx bake fails, try again.

But the yolo job has timeout-minutes: 60. If the first attempt hangs, it burns 60 minutes before the timeout kills it. Now there’s no time budget left for retries 2 and 3. The retry mechanism exists but can never activate, because the failure mode is a hang, not a crash.

A crash fails fast and retries work. A hang fails slow and eats the clock.

The fix is a per-command timeout:

# Instead of relying on job-level timeout for retries
timeout 30m docker buildx bake --push pretraining

# Now a hang gets killed in 30 min, leaving budget for wretry to retry

This way, a hung build gets killed after 30 minutes, the step fails, and wretry can actually use its remaining attempts on a (hopefully different, hopefully healthy) runner.

The cancel-in-progress problem

One more thing. The workflow’s concurrency config has cancel-in-progress: true. This means if you re-run the workflow — or push a new commit to the same PR — the in-flight run gets SIGTERM’d.

On the surface, this makes sense. Don’t waste compute on outdated builds. But in practice, it means a build that was 40 minutes into a 45-minute job gets killed because someone clicked “re-run.” Attempt 2 of my PR run was actually succeeding — 32 cached layers, build progressing normally — and it got murdered by the re-run that created attempt 3.

For long-running build jobs, cancel-in-progress is a footgun. The new run might land on a sick runner and hang, while the cancelled run was about to succeed. You traded a guaranteed finish for a coin flip.

Four fixes, one blind spot

What I’d do:

  1. Per-command timeout on buildx bake — timeout 30m so hangs fail fast and wretry can actually retry. This is the highest-impact change.

  2. Bump the yolo job timeout from 60 to 90 minutes — match rocket_yolo. The current 60-minute timeout leaves zero headroom for retry after a 55-minute hang.

  3. Investigate runner host health — when sequential runner IDs both show timeouts, that’s a host-level problem. The runner infra team needs visibility into which hosts are going flaky. This is the actual root cause; everything else is mitigation.

  4. Get PR builds into Datadog — this was a monitoring blind spot. The yolo amd64 job looked perfect in Datadog because Datadog only had main branch data. The PR failures were invisible. If you’re only monitoring main, you’re only seeing the survivors.

The meta-lesson

The interesting thing about this investigation wasn’t any single finding — it was that neither data source alone would have gotten me there.

GitHub Actions API told me what happened in each run: which step hung, which attempt got cancelled, what the logs said (or didn’t say). But it couldn’t tell me if this was a pattern or a one-off.

Datadog told me the pattern: rocket_yolo fails 19% of the time, always with the same timeout signature, and failures correlate with specific runner hosts. But it couldn’t tell me about my PR build at all, because PR events aren’t indexed.

The runner ID correlation — the actual root cause — was only visible by looking at aggregate failure data with per-event metadata. That’s a Datadog query. But knowing to look at runner IDs in the first place came from seeing the attempt-level details in the GitHub API, where the failure mode (hang vs. crash vs. cancel) pointed away from build logic and toward infrastructure.

Cross-referencing beats staring at one dashboard. Especially for intermittent failures, where the answer is almost never in the logs of the run that failed.