Thirteen shards failed at once. Then a completely different failure showed up wearing the same costume. Then a third thing broke for a third reason. This was last night’s oncall. Here’s what I learned.
1. TLS handshake timeout = check the registry, not your runners
All 13 gpu-tests-falcon shards died at “Initialize containers.” Docker couldn’t pull images from maifalcon.azurecr.io:
net/http: TLS handshake timeout
The data endpoint was maifalcon.westus3.data.azurecr.io. My first instinct was to look at the runner network config. Wrong tree entirely. The ACR team confirmed their auto-scaling had a delay responding to the burst of pulls. They added capacity, problem went away.
TIL: When Docker pull fails with TLS handshake timeout, it’s almost always the registry side. Don’t waste time debugging runner networking. Ask the registry team first.
2. Never assume the same symptom means the same cause
A later run also had falcon GPU test failures. I saw “falcon shards failed” and thought, “ah, the TLS thing again.”
Then I actually checked the logs:
The runner has received a shutdown signal
Exit code 130. That’s runner preemption, not ACR. Completely different problem.
TIL: gh api repos/{owner}/{repo}/actions/jobs/{JOB_ID}/logs and grep for the actual error. Every time. Even if it looks identical to the last failure. Especially then.
3. One run, three failures, three root causes
Run 24161674715 had three failed jobs. I initially treated it as one incident. It was three:
- Falcon shards — runner preemption (exit code 130)
test_sgl_numerics_golden_then_verify—RedisBatchInferenceClienthit aGeneratorExit, a real test bug- cpu-tests gate job — failed because upstream shards were cancelled, not because anything was actually wrong
Each one got a different tag: !dependency, !failed-test, and “not a real failure.” If I’d just looked at the top-level “CI Failed” alert and triaged the whole run as one thing, I’d have mislabeled two out of three.
TIL: When the bot says “Main Required CI Failed,” don’t look at the headline. Open each failed job individually. They’re often unrelated.
4. --no-install-package doesn’t mean --no-import-package
This one was subtle. A CPU-only test (test_rollout_replay_benchmark.py) was getting collected by the redis_gpu_tests job. That job uses --no-install-package mai_kernels to skip the GPU package. Should be fine, right?
Nope. The test imports mai_trainer.evaluation, which imports mai_layers, which imports mai_kernels. The import chain is three levels deep. The package isn’t installed, so the import blows up at collection time, and now your GPU test job is failing on a CPU test for a missing GPU package.
Fix: add --ignore for the test file in the job config (PR #29972).
TIL: If your test job excludes a package at install time, make sure no collected test transitively imports that package. The import chain can be surprisingly deep. --no-install-package is a pip-time concept. Python’s import doesn’t care.
5. Thread hygiene is not optional
By 2am I had six alert threads open. The ones I tagged cleanly (!dependency, !ourfault, !flake, !failed-test) were easy to hand off the next morning. The ones I didn’t tag required re-investigation.
Rules I’m enforcing on myself now:
- Tag every thread with the canonical tag. Even if you’re tired. Especially if you’re tired.
- Link the specific run URL and failed job names. “Falcon failed” is not enough. Which shard? Which run?
- Distinguish “superseded by newer push” (cancelled, not a real failure) from actual failures. These look identical in the alert bot.
- Link back to the main discussion thread so morning-shift people don’t have to search.
Future-you is the first customer of your oncall notes. Write for them.
Nothing here is rocket science. It’s all “read the actual logs” and “don’t assume.” But at 1am with thirteen red shards staring at you, the temptation to pattern-match from the last incident is very real. The antidote is mechanical: open the logs, grep the error, tag the thread. Then move to the next one.