The CI Runner Landscape: Why Your Monorepo Needs More Than ubuntu-latest

If you’ve only ever used ubuntu-latest in your GitHub Actions workflows, your CI setup looks like a single box that runs everything. Lint, test, build, deploy – all on the same ephemeral 2-core VM that GitHub gives you for free. And honestly, that works great. Until it doesn’t.

This post is about what happens when it stops working, and the progression that large monorepos go through as they outgrow the default. It’s a field guide, not a tutorial – the kind of thing I wish someone had shown me when I first looked at a production CI config and saw three different systems stitched together with YAML and duct tape.

This is the compute side of the CI performance problem – where your code actually runs. The cache side is covered in Bazel CI Cache: Why GitHub Actions Cache Won’t Save You.

Level 0: ubuntu-latest and nothing else

GitHub-hosted runners are the default for a reason. Zero infrastructure to manage. No servers to patch, no IAM roles to configure, no disks to monitor. You write a YAML file, push it, and it runs.

runs-on: ubuntu-latest

This covers more than you’d think:

• Linting and formatting – ruff check, prettier, eslint. Fast, stateless, no special access needed.
• Unit tests – as long as they don’t need a GPU or a database with production data.
• Terraform plan – show the diff, don’t apply anything.
• Code review automation – labelers, PR size checks, stale issue bots.
• Documentation builds – Hugo, MkDocs, Sphinx. These are fast.

The constraints

For most repos – and by “most” I mean like 90% of repos on GitHub – these constraints don’t matter. Your Django app doesn’t need VPN access to run pytest. Your TypeScript library doesn’t need 64GB of RAM.

But monorepos are not most repos.

Level 1: self-hosted runners with labels

At some point, you hit a job that ubuntu-latest physically cannot do. Maybe it needs to assume an AWS IAM role. Maybe it needs kubectl access to a cluster that’s behind a VPN. Maybe it’s a CUDA build that needs an actual GPU.

This is when self-hosted runners enter the picture.

The mechanism is simple: you register a machine (or a Kubernetes pod, or an EC2 instance) as a GitHub Actions runner, give it one or more labels, and reference those labels in your workflow:

runs-on: orca

The label is the API. Different labels mean different capabilities. In a real production config, you’ll see patterns like:

Why labels, not just “self-hosted”

You could label everything self-hosted and call it a day. But labels are how you express what a job needs without coupling it to which machine provides it. A job that says runs-on: orca doesn’t care if that runner is an EC2 instance, a K8s pod, or a bare-metal box under someone’s desk. It just needs the capabilities that “orca” promises.

This is also a security boundary. Your linting job doesn’t need AWS credentials. By keeping it on ubuntu-latest, you’re not exposing cloud IAM roles to a workflow that runs prettier. The label system lets you give jobs exactly the access they need and nothing more.

The infra cost

Self-hosted runners are your problem now. You’re managing:

• Provisioning – Who creates the runner? Terraform? Ansible? A bash script in a wiki?
• Scaling – Do you have enough runners for peak load? Too many during off-hours?
• Security – These machines have real credentials. They need patching, monitoring, access control.
• Lifecycle – Ephemeral (spin up per job, tear down after) or persistent (always on, risk of state leaking between jobs)?

Most teams that go down this path end up using actions-runner-controller (ARC) on Kubernetes, which handles scaling and lifecycle automatically. But that’s a whole K8s cluster you’re now operating just to run CI.

Level 2: a second CI system

Here’s where it gets interesting.

At a certain scale, even self-hosted GitHub Actions runners aren’t enough. Not because they can’t run the jobs – they can – but because the teams running the heavy builds need control over the build environment that GitHub Actions’ YAML-based model doesn’t give them.

Enter Buildkite. Or CircleCI. Or Cirrus CI. Or Jenkins (for the old guard). Or Earthly. Or whatever your org picked.

The pattern looks like this: GitHub Actions still runs – it’s still where PR checks report status. But for the heavy compute, a separate CI system handles the actual work. GitHub Actions becomes the orchestration layer, not the execution layer.

Why a whole separate system?

It’s not just about compute. It’s about control.

Teams doing serious C++ compilation, publishing container images, or running end-to-end test suites against distributed systems tend to outgrow what GitHub Actions can express. Not because it’s impossible – you can do almost anything with enough YAML – but because the operational model doesn’t fit.

What this looks like in practice

Look at actual Buildkite queue labels from a production monorepo:

You can’t write runs-on: crow-a100-40gb-8gpu in a GitHub Actions workflow. There’s no label system flexible enough to express “I need 8 A100 GPUs with 40GB VRAM each for an InfiniBand integration test.” This is the kind of hardware targeting that only makes sense with a dedicated CI system that knows its agent fleet intimately.

And these are just the queue labels from one pipeline. A large monorepo might have dozens of pipelines, each with their own queue requirements. The queue label is doing the same thing that runs-on: labels do in GitHub Actions, but the vocabulary is orders of magnitude richer because the CI system manages a fleet of purpose-built machines.

The bridge pattern

This is the part that confused me the first time I saw it. If Buildkite is running the actual build, why is there still a GitHub Actions workflow?

Because GitHub PRs check GitHub Actions status. The green checkmark on your PR comes from GitHub Actions. If your real build runs on Buildkite, you need a way to get that status back to the PR.

The bridge pattern solves this:

# .github/workflows/buildkite-bridge.yml
name: Build
on: [pull_request]

jobs:
  trigger-buildkite:
    runs-on: ubuntu-latest
    steps:
      - name: Trigger Buildkite build
        id: trigger
        run: |
          BUILD=$(curl -s -X POST "https://api.buildkite.com/v2/organizations/$ORG/pipelines/$PIPELINE/builds" \
            -H "Authorization: Bearer ${{ secrets.BUILDKITE_TOKEN }}" \
            -d '{
              "commit": "${{ github.sha }}",
              "branch": "${{ github.head_ref }}",
              "message": "${{ github.event.pull_request.title }}"
            }')
          echo "build_url=$(echo $BUILD | jq -r '.url')" >> $GITHUB_OUTPUT
          echo "build_number=$(echo $BUILD | jq -r '.number')" >> $GITHUB_OUTPUT

      - name: Wait for Buildkite
        run: |
          while true; do
            STATE=$(curl -s "${{ steps.trigger.outputs.build_url }}" \
              -H "Authorization: Bearer ${{ secrets.BUILDKITE_TOKEN }}" \
              | jq -r '.state')
            case $STATE in
              passed) exit 0 ;;
              failed|canceled) exit 1 ;;
              *) sleep 30 ;;
            esac
          done

GitHub Actions triggers a Buildkite build via API, polls until it completes, and exits with the right status code. The PR sees a green or red check. Developers never need to know Buildkite exists (though they usually find out when they need to debug a failure and get redirected to a different UI).

Why not just use Buildkite’s native GitHub integration?

You can. Buildkite has native commit status reporting. But in practice, teams often want:

• Unified status checks – Some checks run on GitHub Actions (lint, format, small tests), others on Buildkite (heavy builds). The bridge keeps everything reporting through the same system.
• Conditional triggering – Maybe you only trigger Buildkite for changes to certain directories. The GitHub Actions workflow can do path filtering before deciding whether to kick off the expensive build.
• Audit trail – GitHub Actions logs show when the external build was triggered and what the result was, all in one place.

What actually replaces actions/cache

Here’s something the runner discussion usually skips: the caching progression that runs in parallel with the compute progression. When you move from GitHub-hosted to self-hosted to Buildkite, you’re not just changing where code runs – you’re fundamentally changing how build artifacts get cached.

The cache post covers the strategies in depth. Here’s what they look like in practice, from a real monorepo.

actions/cache is a rounding error

In this repo’s GitHub Actions workflows, the only caching is setup-python’s built-in pip cache:

- uses: actions/setup-python@v5
  with:
    python-version: "3.11"
    cache: "pip"

That’s it. No actions/cache@v4 for build artifacts. No tarball uploads. The real caching lives entirely outside GitHub Actions, handled by three purpose-built systems.

sccache: Rust/C++ compilation cache → Azure Blob Storage

For compiled languages, the repo uses sccache – Mozilla’s shared compilation cache. It’s like ccache, but with remote storage backends.

Every Buildkite pipeline step runs this at startup:

python -m oaipkg.try_start_sccache

And at the end of each build:

sccache --show-stats

Between those two lines, sccache intercepts every compiler invocation, hashes the inputs, and checks Azure Blob Storage for cached results. Cache hit? Skip the compile. Cache miss? Compile locally, upload the result.

The storage backend is Azure Blob Storage – specifically the rust-sccache container in the oaiartifacts storage account. Auth is via Azure SAS tokens generated at agent startup. This is a remote compilation cache as a service – like a remote cache server, but purpose-built for Rust and C++.

Compare this to what you’d do with actions/cache: compress your entire target/ directory into a tarball, upload 10GB to GitHub’s blob storage, download and decompress it next time. sccache operates at the object file level – individual .o files, not the entire build tree. The granularity difference is like comparing a package manager to “zip up the whole node_modules and pray.”

Buildbarn: Bazel remote cache

For Bazel builds, the repo uses Buildbarn – an open-source remote cache and execution server. This is Bazel’s native --remote_cache approach:

fetch_secret ci/applied/bazel/mac/remote-cache-jwt \
  ~/.openai/ci/baz-remote-cache-jwt.txt

Authenticated via JWT, the Bazel client talks to Buildbarn directly. Every action’s inputs get hashed, the hash is checked against the remote cache, and cache hits skip the build step entirely. This is the Bazel-native answer to “how do I share build results across machines” – and it requires zero GitHub Actions infrastructure.

Persistent EC2 agents: the cache IS the machine

The macOS Buildkite agents are persistent EC2 Mac instances. Not ephemeral. Not spun up per job. They’re always on, and their local disk survives between builds.

From the agent startup script:

BUILDKITE_AGENT_CACHE_HOST_DIR=/Users/ec2-user/ci/cache
BUILDKITE_BUILD_PATH=/Users/ec2-user/ci/builds

That cache directory is the “self-hosted runner with persistent disk” strategy from the cache post, running in production. Zero upload time, zero download time, zero compression. The cache was always there because the machine was always there.

And the startup script does more than just set paths. These machines also run:

• Tailscale VPN connect – so they can reach internal services
• Azure CLI login – so sccache can talk to blob storage
• sccache startup – so compilation caching is ready before the first build step

This is a fully self-contained build environment that boots up with everything it needs. Compare that to a GitHub Actions workflow where the first 10 steps are actions/setup-* for every tool in the chain.

The summary

The punchline: actions/cache handles pip packages. Everything that actually takes time to build – compiled binaries, Bazel actions, C++ object files – uses a purpose-built caching system that GitHub Actions doesn’t even know about.

This is the caching progression that mirrors the compute progression. You don’t just outgrow ubuntu-latest for running builds – you outgrow actions/cache for caching them. And you outgrow it for the same reason: the generic solution works fine until the scale makes the overhead intolerable.

The full picture

Here’s what a mature monorepo’s CI setup actually looks like:

                    ┌─────────────────────────────────┐
                    │          Pull Request            │
                    └──────────────┬──────────────────┘
                                   │
                    ┌──────────────▼──────────────────┐
                    │        GitHub Actions            │
                    │   (orchestration + light jobs)   │
                    └──┬────────┬────────┬────────┬───┘
                       │        │        │        │
              ┌────────▼──┐ ┌──▼─────┐ ┌▼──────┐ ┌▼────────────┐
              │  Lint +   │ │  Unit  │ │ Docs  │ │   Bridge    │
              │  Format   │ │ Tests  │ │ Build │ │   Workflow  │
              │           │ │        │ │       │ │             │
              │ ubuntu-   │ │ubuntu- │ │ubuntu-│ │  ubuntu-    │
              │ latest    │ │latest  │ │latest │ │  latest     │
              └───────────┘ └────────┘ └───────┘ └──────┬──────┘
                                                        │
                                              ┌─────────▼─────────┐
                                              │     Buildkite     │
                                              │  (heavy compute)  │
                                              ├───────────────────┤
                                              │ C++ compilation   │
                                              │ Image publishing  │
                                              │ E2E test suites   │
                                              │ GPU workloads     │
                                              └─────────┬─────────┘
                                                        │
           ┌────────────────────────────────────────────┘
           │
           ▼  Caching layer (not GitHub Actions)
           ┌──────────────────────────────────────────────────┐
           │  sccache ──────────► Azure Blob Storage          │
           │  Buildbarn ────────► Remote cache server         │
           │  Persistent disk ──► Local EBS on EC2 agents     │
           └──────────────────────────────────────────────────┘

        Self-hosted runners (orca, rollout-runner, etc.)
        handle infra ops jobs that need cloud credentials
        or cluster access -- still via GitHub Actions.

Four layers, each doing what it’s good at:

1. GitHub-hosted (ubuntu-latest) – Cheap, fast, stateless jobs. Lint, format, unit tests, docs, and bridge workflows.
2. Self-hosted runners (labeled) – Jobs that need special access: cloud credentials, VPN, GPU, beefy hardware. Still GitHub Actions, but on your machines.
3. External CI system – Heavy compute that needs a different operational model. Dynamic pipelines, custom agents, purpose-built caching, priority queues.
4. Purpose-built caching – sccache, Buildbarn, persistent disk. The caching layer that makes layers 2-3 fast, and that actions/cache was never designed to replace.

The progression

Nobody starts with all four layers. Here’s the typical trajectory:

Stage 1: Everything on ubuntu-latest. Your repo is small. CI takes 5 minutes. Life is good. actions/cache for node_modules works fine.

Stage 2: Add self-hosted runners for specific needs. Someone needs to deploy to a cluster. Someone else needs GPU for testing. You register a few self-hosted runners with labels. Most jobs still run on ubuntu-latest. Caching is still actions/cache, maybe with a bigger key strategy.

Stage 3: Self-hosted runners multiply. More teams, more needs. The runner fleet grows. You set up ARC on Kubernetes. You now have infrastructure to manage infrastructure. Someone notices the persistent disk on self-hosted runners means they don’t need actions/cache anymore.

Stage 4: A separate CI system appears. The team doing C++ builds or ML training needs more control. They set up Buildkite (or similar). Someone writes a bridge workflow. Now you have two CI systems. The caching problem splits: sccache for compiled code, Buildbarn for Bazel, persistent disk for everything else.

Stage 5: GitHub Actions becomes the orchestration layer. Most of the actual compute has moved to self-hosted runners or external systems. GitHub Actions’ primary job is routing: deciding which system should run what, reporting status, and handling the lightweight stuff directly. actions/cache is still there, but only for pip packages – a rounding error in the total caching picture.

Not every monorepo reaches Stage 5. But if you’re looking at a CI config and wondering “why is there a workflow that just… calls another CI system?” – this is why.

Choosing your layer

When you’re adding a new CI job, the decision tree is simpler than it looks:

Does the job need special credentials, network access, or hardware?
├── No  → ubuntu-latest
└── Yes → Do you control the runner fleet?
    ├── No  → You need self-hosted runners (start here)
    └── Yes → Does the job need dynamic pipelines, custom agents,
              or operational control beyond what GHA offers?
        ├── No  → Self-hosted runner with appropriate label
        └── Yes → External CI system + bridge workflow

The key question is never “which CI system is best?” It’s “what does this specific job need that the simpler option can’t provide?” Always start with the simplest layer that works and move up only when you hit a real constraint.

The takeaway

Large monorepos don’t use three CI systems because they love complexity. They use three CI systems because the CI problem has three distinct shapes:

• Stateless commodity compute – ubuntu-latest is unbeatable here.
• Privileged access to infrastructure – self-hosted runners with labels.
• Heavy, customized build pipelines – a dedicated CI system.

And running parallel to all of it, a caching progression that’s just as important: from actions/cache for small stuff, to sccache and Buildbarn for compiled artifacts, to persistent disks that make the cache question disappear entirely.

The bridge pattern ties the compute layers together, and GitHub Actions ends up as the orchestration layer whether you planned for it or not. It’s the thing that has native PR integration, so it’s the thing that reports status, even when the real work happens somewhere else.

If your CI is still entirely on ubuntu-latest and it’s working fine – don’t change anything. Seriously. Every layer you add is infrastructure you operate. But when you start hitting walls – builds too slow, credentials you can’t expose, hardware you don’t have – now you know what the next layer looks like and why it exists.