Your CI run is red. You click into the workflow. You find the test job. You scroll through the logs. You see this:
No failing pytest commands log found
That’s it. The entire output. Your CI failed — somewhere, somehow, some test produced an error — and the reporting system’s best contribution is to tell you that it doesn’t know what happened.
Or worse: you see a list of five failed tests, click into the details, and get the output for… a completely different test than the one that actually caused the cascade. You spend 30 minutes debugging the wrong failure.
This isn’t a testing problem. It’s three logging anti-patterns working together to make CI failure diagnosis maximally painful.
Problem 1: The file that doesn’t exist
The test runner script — run_gpu_tests.py — writes failure information to a file called failing-command-lines.txt. When pytest returns a non-zero exit code, the script appends the failing command to this file. After all test suites finish, a later CI step reads this file and reports the failures.
if return_code != 0:
with open("failing-command-lines.txt", "a") as f:
f.write(f"{pytest_command}\n")
Simple enough. The problem is what happens when the runner doesn’t get that far.
If the runner pod OOMs before the test finishes — which happens regularly with GPU tests — the process is killed mid-execution. The if return_code != 0 check never runs. The file is never written. The reporting step that reads the file finds nothing.
Same thing happens if the Kubernetes node preempts the pod, if the container hits a cgroup memory limit, if NCCL hangs and the job times out at the workflow level, or if a segfault in a native extension kills the Python process.
In all these cases, the test did fail. The runner just died before it could write down that it failed. The CI reporting system sees no file and helpfully reports “No failing pytest commands log found” — a message that communicates exactly zero useful information.
It’s like a black box flight recorder that only works when the plane lands safely.
Problem 2: Last writer wins
When tests do complete (even if they fail), the script captures their output. There’s a second file: last-pytest.out. Every time a pytest suite runs, its output goes here. This file is meant to give developers the actual error messages — stack traces, assertion messages, the stuff you need to diagnose the failure.
The problem is in the name: last-pytest.out.
def run_test_suite(command):
result = subprocess.run(command, capture_output=True)
with open("last-pytest.out", "w") as f: # "w", not "a"
f.write(result.stdout)
return result.returncode
Open mode "w" — write, not append. Each test suite overwrites the previous one’s output.
If the CI job runs three test suites — test_A, test_B, test_C — in sequence, and all three fail:
test_Afails. Output written tolast-pytest.out.test_Bfails. Output overwriteslast-pytest.out.test_A’s output is gone.test_Cfails. Output overwriteslast-pytest.out.test_B’s output is gone.
The developer sees: “test_A, test_B, test_C failed” (from failing-command-lines.txt) but can only read test_C’s output (from last-pytest.out).
Here’s the kicker: in cascading test failures, test_A is usually the root cause. test_B and test_C often fail because they depend on something test_A was supposed to set up, or because the same underlying issue (a broken import, a missing fixture, a configuration error) affects all three. The first failure is the one you want to see. The last failure — the one you actually get — is typically the least informative.
The system is optimized for showing you the wrong failure.
Problem 3: No correlation
These two files — failing-command-lines.txt and last-pytest.out — are the only artifacts produced by the failure reporting system. And they don’t reference each other.
failing-command-lines.txt lists every test suite that failed, one per line:
pytest packages/mai_trainer/tests/test_training.py
pytest packages/mai_distributed/tests/test_nccl.py
pytest packages/mai_layers/tests/test_attention.py
pytest packages/mai_config/tests/test_loading.py
pytest packages/mai_multimodal/tests/test_vision.py
last-pytest.out contains the full output of… one of these. Which one? The last one that ran. Is the order in failing-command-lines.txt the same as the execution order? Probably, but the file doesn’t include timestamps, so you can’t be sure.
Five failures listed. One output visible. No mapping between them.
A developer looking at this has to answer: “which of these five failures should I investigate?” The system gives them the output of one failure — the last one — with no indication of whether it’s the root cause or a downstream casualty. It doesn’t say “this is the output for test_vision.py.” It just says “here’s some pytest output. Good luck.”
The truncation cherry on top
One more thing. The step that reads last-pytest.out and displays it in the CI log does this:
tail -c 131072 last-pytest.out | sed '1d'
tail -c 131072 — take the last 128KB of the file. sed '1d' — delete the first line, because the tail cut might have landed in the middle of a line, producing a partial garbled string.
For most test outputs, 128KB is plenty. But for GPU tests that dump CUDA stack traces, torch error messages, and distributed training logs? 128KB might cut off the beginning of the output — which, in pytest, is where the first failure is reported. The end of the output is the summary line and the last test’s traceback.
So even for the one test whose output you do get, you might be seeing a truncated version that lost the important part.
Why this happens
These aren’t bugs in the traditional sense. Each decision was locally reasonable:
- Writing to a file on failure? Fine. Except the file can’t be written if the process dies.
- Overwriting instead of appending? Keeps the file small. Except you lose all but the last failure.
- Truncating to 128KB? Prevents CI log bloat. Except you lose the beginning of long outputs.
- Separate files for “what failed” and “failure output”? Clean separation. Except there’s no join key between them.
Each compromise is understandable. Together, they create a system where the common case — “multiple tests failed, tell me why” — produces misleading or missing information.
What you actually do
When you hit this in practice, here’s the manual process:
Step 1: Ignore the CI output. Seriously. The displayed output is the last test’s truncated output. It’s almost never what you need.
Step 2: Download the full artifacts. GitHub Actions preserves the raw files as workflow artifacts. Go to the workflow run → Artifacts → download test-results or failing-tests (whatever the artifact name is). Open failing-command-lines.txt to see ALL failures. Open the full pytest output files (if they were uploaded individually) to see each test’s complete output.
Step 3: Start from the top. Look at the first failure in the list, not the last. If you can get its output, start there. If you can’t (because only last-pytest.out was captured), re-run just that one test locally.
Step 4: Check for OOM. If failing-command-lines.txt doesn’t exist at all, check the runner pod’s exit code and Kubernetes events. If the pod was OOM-killed, the failure isn’t a test bug — it’s a resource allocation problem.
This workflow takes 5-15 minutes of artifact downloading and log archaeology. Multiply by 12 failures per day. Multiply by the team. That’s a lot of hours spent navigating a logging system that’s actively pointing you in the wrong direction.
What the fix looks like
The right fix isn’t complicated. It’s just work that hasn’t been prioritized because “CI output is bad” competes with “features customers pay for.”
Fix Problem 1: Write a sentinel file before tests start, not after they fail. If the sentinel exists but no results file exists, the runner died mid-test. Report that explicitly: “Runner died during test execution — check pod events for OOM or preemption.” Now the absence of data is data.
Fix Problem 2: Append, don’t overwrite. Or better — write one output file per test suite, named after the suite. test_training.out, test_nccl.out, test_attention.out. Upload all of them as artifacts. No overwriting, no information loss.
Fix Problem 3: Structured output. Instead of two unrelated files, write a single JSON or JSONL file:
{"suite": "test_training.py", "status": "FAIL", "duration": 34.2, "output_file": "test_training.out"}
{"suite": "test_nccl.py", "status": "FAIL", "duration": 12.7, "output_file": "test_nccl.out"}
{"suite": "test_attention.py", "status": "FAIL", "duration": 8.1, "output_file": "test_attention.out"}
Now every failure is linked to its output. The reporting step can show a table with all failures and their first error line. A developer knows exactly where to look and in what order.
Fix the truncation: Don’t truncate for CI display. If the log is too long for GitHub’s UI, that’s what the artifact download is for. Show the first 64KB (where the first failure is) instead of the last 128KB (where the last failure’s summary is).
None of these are hard. They’re just invisible until you’ve spent enough mornings staring at “No failing pytest commands log found” to want to fix the meta-problem instead of the test problem.
CI failure reporting is infrastructure that everyone uses and nobody owns. It works well enough that it doesn’t trigger an incident, and badly enough that it wastes hours every week. The uncanny valley of tooling — too broken to trust, too functional to replace.