The 500 Lines That Run Every GPU Test

A nightly test got cancelled at exactly 6 hours. No error, no stack trace, no Slurm failure. Just… cancelled. The kind of CI failure where you stare at the logs and find nothing, because the thing that killed your job isn’t in your logs.

Yipu tracked it down in PR #22547. Seven lines changed. The fix is boring. But the thing it fixes — and the infrastructure it lives in — is not.

The timeout that ate a nightly run

The job was daily_mai_text_v1_5b_tests. It trains a 5B parameter model on 64 GPUs across 8 nodes. Expected runtime: about 6-7 hours. The SSH timeout was set to 800 minutes — plenty of headroom.

But the job kept dying at exactly 360 minutes.

Here’s the thing: there are two independent timeout mechanisms in play, and they don’t know about each other.

Timeout 1: SSH command_timeout. This is how long the SSH action waits for your script to finish on the remote host. Set by the timeout input. For this job: 800 minutes. Generous.

Timeout 2: GitHub Actions timeout-minutes. This is a job-level timeout that GitHub enforces. Default: 360 minutes. Six hours. And nobody had set it.

So the SSH action was willing to wait 800 minutes. But GitHub killed the entire job after 360. The SSH connection just… vanished. From Slurm’s perspective, the job is still running. From GitHub’s perspective, the workflow is cancelled. From the engineer’s perspective, something is broken and there’s nothing useful in the logs.

The fix is seven lines: add a timeout_minutes input to the reusable workflow, wire it to timeout-minutes on the job, and update the four callers that exceed 360 minutes. Done.

But the interesting question isn’t the fix. It’s: what is this reusable workflow, and why does a single file control every GPU test in the repo?

What s_commands actually is

s_commands.yml is a ~500-line reusable GitHub Actions workflow. Every GPU test, every daily nightly run, every multi-node training validation in our monorepo goes through this one file. Four workflow files call it. About 20 jobs total.

It exists because of a fundamental constraint: GitHub Actions runners can’t run GPU code.

Our CI runs on GitHub Actions. Our GPU jobs run on Slurm — a job scheduler designed for HPC clusters, managing hundreds of GPUs across dozens of nodes. These two systems live in completely different worlds. GitHub Actions gives you a CPU VM in the cloud. Slurm gives you GPU nodes in a data center. They share nothing except a network connection and an NFS mount.

s_commands is the bridge. Here’s what it does:

GitHub Actions runner (CPU VM)
  1. Checks out the repo
  2. Tars the entire repo to /mnt/ci/gha (shared NFS mount)
  3. SSHes into slurm-login-1.tenant-slurm
  4. Extracts the tar on the Slurm node
  5. Sets up the Python environment
  6. Runs the caller's script (which submits sbatch/srun GPU jobs)
  7. Collects artifacts via shared /mnt/ci/artifacts mount
  8. Cleans up workdir + tar
  9. Reports success/failure back to GitHub

That’s it. Tar, SSH, run, collect, clean up. The entire GPU CI capability of a 100+ package monorepo flows through this pattern.

Why SSH and tar?

If you’re coming from a world where CI runners are your compute, this looks weird. Why not just run on the GPU nodes directly?

Because Slurm doesn’t work that way. You don’t SSH into a GPU node and run code. You submit a job to a scheduler, which allocates resources from a shared pool, and runs your code when resources are available. The login node is the only stable entry point — it’s where you sbatch or srun from.

And GitHub Actions runners can’t see the Slurm cluster directly. They’re VMs in a different network segment. The only shared surface is:

1. SSH access to the Slurm login node (via appleboy/ssh-action)
2. NFS mounts visible to both the CPU runners and the Slurm nodes (/mnt/ci/gha for staging, /mnt/ci/artifacts for output)

So the pattern becomes: stage your code on NFS, SSH in, extract it, run it, collect the output from NFS. It’s the simplest thing that could possibly work.

The tar step is surprisingly important. You can’t just git clone on the Slurm node — the ci user on the Slurm login node doesn’t have repo access, and you want to test the exact commit that triggered CI, not whatever main happens to be. Tarring the checkout from the GitHub runner and extracting it on the Slurm side guarantees the code is identical.

The 13 knobs

s_commands has 13 inputs. That sounds like a lot, but each one exists for a reason — they’re the accumulated wisdom of “what goes wrong when you run GPU tests from CI.”

The interesting ones:

use_srun — wraps your script in srun, which requests an interactive Slurm allocation. Some tests need to run directly on a GPU node (interactive pytest). Others submit their own sbatch jobs and just need the login node. This toggle handles both.

use_local_workdir — uses /tmp on the Slurm node instead of the shared NFS mount. Faster extraction, but the data is node-local. Good for tests that don’t need to share state across nodes.

setup_env — four modes: standard (full uv sync with system torch), manual (manual torch install), manual_system (manual torch with system site-packages), or none. Different tests need different Python environments, and this is cheaper than building different Docker images.

cancel_job_prefix — before running, SSH in and scancel any Slurm jobs matching this prefix. Prevents stale GPU jobs from lingering when a workflow is re-triggered. Without this, you’d accumulate zombie GPU allocations every time a nightly re-runs.

retries — retry count for the script. Sounds simple. The implementation is not.

The subprocess trick

Here’s my favorite bit of s_commands. The retry mechanism.

The script the caller provides runs inside a shell. You’d think retrying a failed script is trivial — just run it again. But there’s a subtle problem with set -e.

If you eval a script that has set -e inside it, and something fails, set -e kills the entire shell — not just the eval’d script. Because eval doesn’t create a subshell. It runs in the current shell context. So the exit code never gets captured. The retry logic never runs. Your “retry 3 times” is actually “retry 0 times and die.”

The fix: write the script to a temp file and run it as a subprocess with bash "$TEMP_SCRIPT". Now set -e inside the script kills the subprocess. The parent shell captures the exit code via $?. Retries actually work.

# This breaks retries:
eval "$MAIN_SCRIPT" || EVAL_RC=$?    # set -e kills the parent shell

# This works:
bash "$TEMP_SCRIPT" || EVAL_RC=$?    # set -e kills only the child

The comment in the actual code explains this. It’s one of those things where the comment is more valuable than the code — without it, some future engineer would “simplify” it back to eval and break every retry in CI.

Failure detection by absence

How does s_commands know if a job timed out versus failed versus succeeded?

It doesn’t check the SSH exit code. Or rather — it can’t only check the exit code, because a timeout looks the same as a killed SSH connection looks the same as a bunch of other things.

Instead, the script writes a status file:

# On success (inside the script):
echo "no_error" > "$STATUS_FILE"

# On failure (in the retry wrapper):
echo "error" > "$STATUS_FILE"

Then, after the SSH step:

if [ -f "$STATUS_FILE" ]; then
    SCRIPT_STATUS=$(cat "$STATUS_FILE")
    echo "failure_type=$SCRIPT_STATUS"
else
    # File not written = timeout
    echo "failure_type=timeout"
fi

If the status file says “no_error” — success. If it says “error” — the script failed. If the file doesn’t exist — the SSH connection was killed before the script could write anything. That’s a timeout.

Detection by absence. The signal is the missing file. There’s something elegant about it — you don’t need a heartbeat protocol or a sidecar process. If the file exists, someone was alive long enough to write it. If it doesn’t, they weren’t.

This is also how Yipu’s timeout bug was detectable. The job was killed by GitHub’s 360-minute limit, which severed the SSH connection, which meant the status file was never written. failure_type=timeout. Except nobody had configured the timeout to be that, so it looked mysterious until you realized there were two timeouts and one was a default you never set.

The two-timeout lesson

This is the generalizable takeaway: when two systems each have their own timeout, the shorter one wins, and it might not be the one you configured.

In s_commands:

• SSH command_timeout: set explicitly per job
• GitHub Actions timeout-minutes: not set, defaulting to 360

The callers set the SSH timeout to 400, 600, 800 minutes. They thought they were controlling how long the job runs. They were controlling how long SSH waits. But GitHub was controlling how long the job lives. And GitHub’s default was shorter.

You see this pattern everywhere:

Six layers. Each with its own timeout. The lowest one wins. And the one you forgot to set is always the one that fires.

The fix is always the same: when you set a timeout at one layer, check what’s above and below. Set them all. Make them consistent. And when a job dies at a suspiciously round number — 360 minutes, 6 hours, 3600 seconds — it’s almost certainly a default timeout somewhere in the stack.

Who calls this thing

Four workflows, about 20 jobs:

daily_tests.yml — the big one. ~14 jobs covering integration tests, performance regression, multi-node training (5B, 20B models), vision encoder training, image generation, RM training, evaluator tests. These run every night at 2 AM Pacific. Some take 30 minutes. Some take 8 hours. All go through s_commands.

rocket_regression_tests.yml — 5 jobs for the Rocket inference engine. Performance benchmarks, correctness checks.

gpu_integration_tests.yml — 2 jobs for PR-triggered GPU tests. The gatekeepers before merge.

run_sgl_eval_trigger.yml — 1 job for on-demand model evaluation.

Every one of them has the same shape: a uses: ./.github/workflows/s_commands.yml line, a timeout, a script, and maybe some knobs. The script is the part that varies — it might be a single pytest command, or a 30-line training launcher invocation with 40 flags. But the infrastructure to get that script onto a Slurm node and get results back? That’s always s_commands.

The invisible infrastructure

s_commands is one of those files that nobody thinks about until something breaks. It’s not in any onboarding doc. It’s not in any architecture diagram. There’s no README that says “hey, this 500-line YAML file is the single gateway between your CI system and your GPU cluster.”

It just works. And the 7-line timeout fix that Yipu shipped is the kind of change that reveals the whole architecture — a small crack you peer through and see the machinery behind the wall.

If you’re building something similar — connecting GitHub Actions to an HPC cluster, or any CI system to compute that lives on a different network — the pattern is transferable. SSH bridge, shared filesystem for staging, subprocess for retry isolation, status files for failure detection, explicit timeouts at every layer.

And if your job dies at exactly 6 hours for no apparent reason, check the defaults.