Wait-for-Label: Gating Expensive CI Jobs Behind a Human Decision

GPU runners are scarce. Every PR in our monorepo triggers CI, and most of those PRs don’t need GPU tests — they’re touching configs, fixing typos, updating documentation. But the GPU test jobs don’t know that. They see a PR event, they spin up, they burn 8xH100 time running tests against code that changed a README.

So we built a gate. A job called wait-for-label that sits between “PR pushed” and “GPU tests run,” polling for a human to say “yes, this PR actually needs GPUs.” It’s a simple idea. The implementation touches every weird corner of GitHub Actions job orchestration.

The shape of the problem

Our CI workflow has jobs that are cheap (linting, type-checking, CPU tests) and jobs that are expensive (GPU tests on Slurm clusters). The cheap jobs should run on every push. The expensive jobs should only run when someone decides they’re necessary.

GitHub Actions doesn’t have a native “wait for human approval” primitive for PR workflows. There’s environment protection rules for deployments, but those don’t fit the PR CI model. What we want is: push a PR, have cheap tests run immediately, and hold the expensive tests until a human adds a label.

The mechanism: a job that polls the GitHub API every 2 minutes, checking if a specific label (run-gpu-tests) exists on the PR. Downstream GPU jobs declare a needs: dependency on this polling job. They literally can’t start until the poller succeeds.

GitHub Actions `needs:` — the dependency you can’t escape

Before we look at the code, you need to understand how needs: works in GitHub Actions. It’s the core mechanism that makes the entire pattern possible.

jobs:
  setup:
    runs-on: ubuntu-latest
    steps:
      - run: echo "setting up"

  tests:
    needs: [setup]
    runs-on: ubuntu-latest
    steps:
      - run: echo "testing"

needs: creates a hard dependency. tests will not start until setup completes. If setup fails, tests is skipped. If setup is cancelled, tests is skipped. If setup succeeds, tests evaluates its own if: condition and decides whether to run.

This is the lever. Make wait-for-label a job. Make GPU tests needs: [wait-for-label]. Now GPU tests literally cannot run until the label-polling job finishes.

Here’s how it’s wired in run_gpu_tests.yml:

jobs:
  gpu_tests:
    needs: [find_changed, build_image, calculate_shards, wait-for-label]
    if: >-
      ${{ (needs.find_changed.outputs.has_tests == 'true'
           && needs.calculate_shards.outputs.shard_count > 0) }}

  redis_gpu_tests:
    needs: [find_changed, build_image, wait-for-label]
    if: >-
      ${{ inputs.cluster != 'falcon-phx-gb'
          && (needs.find_changed.outputs.mai_distributed_changed == 'true' || ...) }}

Both gpu_tests and redis_gpu_tests list wait-for-label in their needs: array alongside the other dependencies they actually need outputs from (find_changed, build_image, etc.). The wait-for-label job doesn’t produce any outputs that these jobs consume. It’s purely a gate — its only purpose is to block execution until it succeeds.

Job-level `if:` vs step-level `if:` — this matters

Here’s a subtlety that’s central to how the pattern works, and it’s the kind of thing that bites you if you don’t understand it.

Job-level if: — if the condition is false, the job is skipped. It never gets a runner. Its status is skipped.

Step-level if: — the job still runs and gets a runner. The step with the false condition is skipped, but the job itself completes with status success (assuming no other steps fail).

This distinction matters because of how needs: handles different statuses:

Upstream status	Downstream behavior (default)
`success`	Downstream runs normally
`failure`	Downstream is skipped
`cancelled`	Downstream is skipped
`skipped`	Downstream is skipped

See the last row? A skipped job causes its dependents to skip too. This is the default behavior, and it cascades. If A is skipped, B (which needs A) is skipped, and C (which needs B) is skipped, all the way down.

The wait-for-label job uses this to its advantage. In the original design, the guards are at the step level:

jobs:
  wait-for-label:
    runs-on: cpu-tiny-falcon-ca
    timeout-minutes: 4320
    needs: [find_changed]
    steps:
      - name: Checkout code
        uses: actions/checkout@v4
        if: inputs.caller == 'pr-ci' && needs.find_changed.outputs.has_tests == 'true'
      - name: Wait for "run-gpu-tests" label
        if: inputs.caller == 'pr-ci' && needs.find_changed.outputs.has_tests == 'true'
        uses: ./.github/actions/block-for-label
        with:
          pr-number: ${{ github.event.pull_request.number }}

Both steps have if: inputs.caller == 'pr-ci' && .... On merge queue or main branch pushes, caller isn’t pr-ci, so both steps are skipped. But the job itself still runs — it gets a runner, skips all its steps, and completes with success.

Why does this matter? Because a success status propagates cleanly through needs:. The downstream GPU jobs see a successful dependency and proceed to evaluate their own if: conditions. If the guards were at the job level instead, the job would be skipped on non-PR contexts, which would cascade-skip all GPU tests even on merge queue runs where you definitely want them.

The polling mechanism

The actual label-checking is a composite action at .github/actions/block-for-label/action.yml. Let’s trace through it.

Step 1: Initial probe. Before entering the polling loop, check if the label already exists:

LABEL_PRESENT=$(gh api repos/${{ github.repository }}/pulls/${{ inputs.pr-number }} \
  --jq '.labels[].name' | grep -x '${{ inputs.label }}' || true)
if [[ -n "$LABEL_PRESENT" ]]; then
  echo "label_found=true" >> "$GITHUB_OUTPUT"
else
  echo "label_found=false" >> "$GITHUB_OUTPUT"
fi

This is a fast path. If someone adds the label before pushing (or the label persists from a previous push), skip the polling entirely.

Step 2: Post a comment. Depending on whether the label was found, post either a “GPU tests enabled” or “waiting for label” comment on the PR. The comment_tag: gpu-tests-block-status ensures subsequent updates replace the same comment instead of creating new ones.

Step 3: Poll. If the label wasn’t found initially, enter the polling loop:

const prNumber = ${{ inputs.pr-number }};
const labelToWaitFor = '${{ inputs.label }}';
const waitTimeoutMinutes = parseInt(process.env.WAIT_TIMEOUT_MINUTES, 10);
const stopTime = Date.now() + (waitTimeoutMinutes * 60 * 1000);

let attempt = 0;
const delay = ms => new Promise(res => setTimeout(res, ms));

while (Date.now() < stopTime) {
  attempt++;
  try {
    const { data: pr } = await github.rest.pulls.get({
      owner: context.repo.owner,
      repo: context.repo.repo,
      pull_number: prNumber,
    });

    const labels = pr.labels.map(label => label.name);
    if (labels.includes(labelToWaitFor)) {
      console.log(`Required label '${labelToWaitFor}' found`);
      core.setOutput('label_found', 'true');
      return;
    }
  } catch (error) {
    console.error(`Error fetching PR #${prNumber}:`, error);
  }

  const remainingMinutes = Math.ceil((stopTime - Date.now()) / (60 * 1000));
  console.log(`Label not found. Attempt ${attempt}. ${remainingMinutes} min remaining.`);
  await delay(120000); // 2 minutes
}

console.log(`Timeout reached after ${waitTimeoutMinutes} minutes.`);
core.setOutput('label_found', 'false');

This runs inside actions/github-script@v6, which gives you the github client (authenticated Octokit instance) and core (for setting outputs and logging). The loop is straightforward: call pulls.get(), check labels, sleep 2 minutes, repeat. On timeout, it sets label_found to false and lets the job succeed anyway — the 3-day timeout is a safety valve, not a hard rejection.

Step 4: Post-poll comment. After the loop exits, update the PR comment to reflect what happened — label found, or timeout reached.

The whole thing is a composite action, meaning these steps run in the caller’s context on the caller’s runner. There’s no separate job or container — the steps are inlined into whatever job calls uses: ./.github/actions/block-for-label.

The runner trick: why self-hosted?

Here’s a constraint that’s easy to miss: GitHub-hosted runners have a 6-hour job timeout. That’s a hard limit. You can set timeout-minutes: 4320 all you want — GitHub will kill your job at 360 minutes on a hosted runner.

But we want to wait up to 3 days. A PR might get pushed on Friday afternoon, and the developer won’t add the run-gpu-tests label until Monday.

The solution: run the polling job on a self-hosted runner. Self-hosted runners don’t have the 6-hour limit — the timeout is whatever you configure (up to 35 days, per GitHub’s docs).

wait-for-label:
  runs-on: cpu-tiny-falcon-ca
  timeout-minutes: 4320  # 3 days

cpu-tiny-falcon-ca is a purpose-built runner type — a fraction of a CPU core and 2GB of RAM. It’s a tiny VM whose only job is to sit there and poll an API every 2 minutes. We can run many of these on a single physical machine, so the cost is negligible.

This is the kind of infra decision that’s invisible until you hit the constraint. “Why don’t you just use a GitHub-hosted runner?” Because it dies after 6 hours. “Why don’t you use your GPU runners?” Because those are scarce and expensive and should be running tests, not sleeping. So you build a new runner class — the smallest VM that can run curl — and you dedicate it to waiting.

Composite actions: the reuse mechanism

The block-for-label action is defined as a composite action in .github/actions/block-for-label/action.yml. This is worth explaining because composite actions have a specific execution model that differs from reusable workflows.

A composite action is a YAML file with runs: using: "composite" and a list of steps. When a job calls uses: ./.github/actions/block-for-label, those steps get inlined into the calling job. They run on the same runner, in the same workspace, with access to the same environment.

runs:
  using: "composite"
  steps:
    - name: Create GitHub App token
      id: create-app-token
      if: ${{ inputs.enable-app-token == 'true' && inputs.app-id != '' && inputs.private-key != '' }}
      uses: tibdex/github-app-token@v2
      with:
        app_id: ${{ inputs.app-id }}
        private_key: ${{ inputs.private-key }}

    - name: Initial label probe
      id: initial_probe
      shell: bash
      # ...

    - name: Wait for required label
      if: steps.initial_probe.outputs.label_found == 'false'
      uses: actions/github-script@v6
      # ...

Key things about composite actions:

Inputs are defined in the inputs: block of action.yml and referenced as ${{ inputs.foo }}
Steps can reference each other via steps.<id>.outputs.<name>, same as a regular job
Shell must be explicit — composite steps require shell: bash (regular workflow steps inherit from the runner)
The caller must checkout the repo first if the composite action is at a local path (./.github/actions/...)

That last point explains why the wait-for-label job has a checkout step with sparse-checkout:

- name: Checkout code
  uses: actions/checkout@v4
  with:
    fetch-depth: 1
    sparse-checkout: |
      .github/actions/block-for-label

It only checks out the directory containing the composite action — not the full repo. Minimal checkout for a job whose only purpose is to poll an API.

The authentication dance

The action supports two authentication modes. This is a subtle but important detail for anyone implementing a similar pattern.

Mode 1: GitHub App token. If app-id and private-key are provided and enable-app-token is true, the action generates a short-lived token via tibdex/github-app-token@v2:

- name: Create GitHub App token
  id: create-app-token
  if: ${{ inputs.enable-app-token == 'true' && inputs.app-id != '' && inputs.private-key != '' }}
  uses: tibdex/github-app-token@v2
  with:
    app_id: ${{ inputs.app-id }}
    private_key: ${{ inputs.private-key }}

Mode 2: Default GITHUB_TOKEN. If no app credentials are provided, it falls back to the workflow’s default GITHUB_TOKEN.

Every subsequent step uses a fallback expression for the token:

env:
  GITHUB_TOKEN: ${{ steps.create-app-token.outputs.token || github.token }}

The || operator means: use the app token if it exists, otherwise fall back to github.token. This is a clean pattern — the action works in both configurations without conditional logic in every step.

Why bother with a GitHub App at all? The default GITHUB_TOKEN has limited permissions and scoping. A GitHub App token can be configured with exactly the permissions you need (in this case: read PR labels, write PR comments) and works across repos if needed. It also avoids the problem where GITHUB_TOKEN events don’t trigger subsequent workflow runs.

The three messages

The action posts one of three comments on the PR, depending on what happens:

Waiting:

GPU tests have been disabled on this PR to conserve scarce GPU resources. Once you have type-checks and CPU tests passing, you can enable GPU tests by setting the “run-gpu-tests” label.

Label found:

This PR carries the run-gpu-tests label. All pushes to this PR will run GPU tests. To change that, remove the label.

Timeout:

Waiting for the “run-gpu-tests” label to be added to this PR has timed out after 3 days. GPU tests will be run on this commit, but future pushes will not run GPU tests unless the label is added.

All three use the same comment_tag: gpu-tests-block-status. The thollander/actions-comment-pull-request@v2 action uses this tag to find and update an existing comment instead of posting a new one. So the comment transitions: waiting → enabled (or waiting → timeout), always in the same comment. No spam.

The timeout behavior is worth noting: after 3 days, the job succeeds anyway and GPU tests run. This is a safety valve — if someone forgets to add the label, the tests still run eventually. The cost model says “wasting GPU time after 3 days is better than blocking a PR forever.”

The `always()` pattern

Here’s where it gets interesting. The basic pattern — needs: [wait-for-label] — works great when wait-for-label always runs. But what if you want wait-for-label to be skipped in some contexts?

Remember: a skipped job cascades through needs:. If wait-for-label is skipped, every job that depends on it is also skipped. That’s fine for PR pushes where the condition should gate. It’s terrible for merge queue runs where you want GPU tests to run unconditionally.

The original design avoids this problem by keeping the guards at the step level — the job always runs, steps are conditionally skipped, and the job succeeds. But there’s a cost: even on merge queue runs where no polling is needed, the job claims a runner, does nothing, and releases it. Tiny runner, tiny cost, but it adds latency and it’s just… wasteful.

The fix is always(). From the ST tests workflow (run_st_tests.yml), which already implements this:

st_checkpoint_tests:
  needs: [find_changed, build_st_base, wait-for-label]
  if: >-
    always() && !cancelled() &&
    needs.build_st_base.result == 'success' &&
    (inputs.run_all == true || needs.find_changed.outputs.st_core == 'true' || ...)

always() is a status function that changes how if: conditions interact with needs:. Normally, if any job in needs: is skipped or failed, the downstream job doesn’t even evaluate its if:. With always(), the if: is always evaluated, regardless of upstream status.

But always() alone is too broad — it also runs on cancellation, which you almost never want. So the pattern is always() && !cancelled(). This means: evaluate my if: condition even if upstream jobs were skipped or failed, but don’t run if the workflow was cancelled.

Then you add explicit result checks:

needs.build_st_base.result == 'success'

This is the always() pattern: opt out of automatic skip cascading, then manually check the results you care about. You’re saying: “I know wait-for-label might be skipped, and that’s fine — I only care that build_st_base succeeded.”

The available result values:

Value	Meaning
`'success'`	Job completed successfully
`'failure'`	Job failed
`'cancelled'`	Job was cancelled
`'skipped'`	Job was skipped (job-level `if:` was false, or a dependency was skipped)

Without always(), you never get to check these — the downstream job is skipped before its if: is evaluated.

The optimization: PR #23239

There’s an active PR (#23239) that evolves the pattern. The insight: why run wait-for-label at all on merge queue or main branch? Move the guard from step-level to job-level:

wait-for-label:
  if: inputs.caller == 'pr-ci' && needs.find_changed.outputs.has_tests == 'true'
  runs-on: cpu-tiny-falcon-ca
  timeout-minutes: 4320
  needs: [find_changed]

Now on merge queue: the job is skipped (not success with skipped steps). No runner claimed. No latency added. Zero cost.

But this means downstream jobs need the always() pattern:

gpu_tests:
  needs: [find_changed, build_image, calculate_shards, wait-for-label]
  if: >-
    always() && !cancelled() &&
    needs.build_image.result == 'success' &&
    needs.calculate_shards.result == 'success' &&
    (needs.find_changed.outputs.has_tests == 'true'
     && needs.calculate_shards.outputs.shard_count > 0)

The tradeoff: the if: condition gets more verbose because you have to explicitly check the results of every dependency you actually care about. But the semantics are cleaner — wait-for-label only runs when it’s doing real work, and downstream jobs explicitly handle the skipped case.

This is a natural evolution. The original design optimized for simplicity (step-level guards, always-succeeding job). The new design optimizes for correctness and resource efficiency (job-level guard, explicit result handling). You tend to arrive at the second design after living with the first one for a while and noticing the wasted runners in your CI dashboards.

Putting it all together

Here’s the full flow, end to end:

1. Developer pushes to PR
2. CI workflow triggers
3. find_changed runs → determines which tests exist
4. wait-for-label job starts on cpu-tiny-falcon-ca
   a. Checks if "run-gpu-tests" label exists → if yes, succeeds immediately
   b. If no → posts "waiting" comment, enters 2-minute polling loop
   c. Human adds label → poll detects it → posts "enabled" comment → succeeds
   d. 3 days pass → posts "timeout" comment → succeeds anyway
5. gpu_tests and redis_gpu_tests see wait-for-label succeed
6. They evaluate their own if: conditions
7. GPU tests run on Slurm cluster

On merge queue or main branch (with the PR #23239 optimization):

1. Merge queue event triggers
2. find_changed runs
3. wait-for-label is skipped (job-level if: is false)
4. gpu_tests uses always() to evaluate despite skipped dependency
5. gpu_tests checks build_image.result == 'success', etc.
6. GPU tests run unconditionally

The key concepts, in order of how likely you are to get bitten by them:

needs: skipping cascades. A skipped dependency skips everything downstream. This is the default and it’s aggressive.
Job-level if: produces skipped. Step-level if: produces success. These are not the same thing, and needs: treats them differently.
always() && !cancelled() is the escape hatch. It lets you evaluate your if: even when a dependency was skipped.
Self-hosted runners bypass the 6-hour limit. If you need a job that waits for days, you need your own runners.
Composite actions run in the caller’s context. They’re inlined steps, not separate jobs. The caller must checkout the repo for local actions to resolve.

The pattern itself — polling for a label to gate expensive work — is simple. The GitHub Actions machinery required to make it work correctly across PR, merge queue, and main branch contexts is where all the complexity lives. Every piece exists because someone hit a wall with the simpler version first.