CI Observability: Runner Utilization and the Wrong Question

We run 56,000 CI jobs a day across a dozen runner pools. Somebody asked the obvious question: how busy are these runners? Reasonable question. Turns out the answer — and the real question — are both weirder than you’d expect.

The shape of the problem

Our runners are ephemeral. One job, one instance, gone. There’s no machine you can SSH into and run htop. You can’t check CPU usage because the CPU didn’t exist five minutes ago and won’t exist five minutes from now.

What you can do is look at the CI events flowing into Datadog. Every job has a start time, a duration, a runner label, and a queue time. So the question becomes: given a stream of (start, duration) tuples, how many jobs are running concurrently at any given moment?

This is the “overlapping intervals” problem. Every CS grad has solved it on a whiteboard. Datadog has… not solved it.

56K jobs, 12 pools, very unequal

Before we get into the visualization rabbit hole, here’s what a day of CI actually looks like:

The yolo-ubuntu-latest pool handles 23K jobs but each one takes 10 seconds — it’s lint checks and fast validations. The big pool runs fewer than 2K jobs but they average 6 minutes each, consuming 191 hours of compute in a 24-hour window. That’s 8 days of wall-clock compute squeezed into one calendar day.

So “how busy” depends entirely on whether you mean “how many jobs” or “how much compute.”

Datadog can’t count overlapping intervals

Here’s the thing about Datadog CI analytics — it stores events, not time series. Each job is a point with a start time and a duration. To know how many jobs overlap at time T, you’d need to check every job where start <= T < start + duration.

DD doesn’t do that. There’s no native “count overlapping intervals at each point” function. Not in dashboards, not in notebooks, not in the query language.

The standard workaround is the cumsum trick: bucket your timeline (say, 1-minute windows), count how many jobs started in each bucket, count how many ended in each bucket, then compute the cumulative sum of starts minus cumulative sum of ends. At any bucket, that running total approximates concurrent jobs.

concurrency[t] ≈ cumsum(starts[0..t]) - cumsum(ends[0..t])

It works. Mostly. The cumulative sums accumulate rounding errors across bucket boundaries — a job ending at 14:59:59.9 gets bucketed differently than one ending at 15:00:00.1, and that drift adds up over 24 hours. For a rough “are we at 5 or 50 concurrent jobs” it’s fine. For capacity planning with precision, it’s not great.

Someone already built the dashboard

Plot twist: halfway through this investigation, I found a Runner Infra dashboard that already existed. It had the cumsum approach implemented, plus some queue time panels.

This is the DevEx experience in a nutshell. You spend two hours reinventing something, then discover someone on another team built it six months ago. The dashboard wasn’t wrong — it just wasn’t answering the question I actually needed answered.

The script approach

I also wrote a Python script that pulls raw events from the DD API and computes exact concurrency. No bucketing, no cumsum drift — just sort all start and end timestamps, sweep through them, track the running count.

For the big runner pool on one day: peak 20 concurrent jobs at 16:57 UTC. The API returns 1,000 events per page with cursor pagination, so a full day of big jobs takes about 2 pages. The yolo-ubuntu-latest pool? 57 pages. API rate limits start mattering real fast.

The script gives you exact numbers but doesn’t scale to a live dashboard. It’s an audit tool, not a monitoring solution.

Queue time is the real metric

Here’s the thing I should have realized at the start: concurrency doesn’t tell you what you actually want to know.

Knowing that we peak at 20 concurrent big jobs is… interesting? But what do you do with that? Is 20 too many? Too few? Right-sized? You can’t tell from concurrency alone. You’d need to know the pool’s capacity limit, compare it to the peak, and even then — if the peak only lasts 30 seconds, does it matter?

Queue time answers the real question: are developers waiting?

• High concurrency + low queue time = pool is right-sized, you’re using what you have
• High concurrency + high queue time = pool is undersized, people are blocked
• Low concurrency + low queue time = pool might be oversized (or traffic is just low)

P95 queue time by runner label, trended over time, is the metric that actually drives scaling decisions. And DD already stores queue time on every event — @ci.queue_time in nanoseconds. No cumsum tricks, no bucket drift, no 57-page API pagination. Just a straightforward percentile aggregation.

The concurrency visualization is cool. The queue time chart is useful.

What I’d actually build

If I were setting up runner monitoring from scratch:

• Primary signal: P95 queue time by runner label, 7-day rolling, with alert at some threshold (30s? 60s? depends on the pool)
• Secondary signal: Job count by label over time — not for capacity, but for anomaly detection. If big jobs suddenly triple, something changed upstream
• Nice-to-have: The concurrency chart, for gut-checking during incidents. “Are we actually saturated or is something else going on?”
• Skip: Exact real-time concurrency. The effort-to-insight ratio is terrible for a live dashboard

The concurrency problem is a fun puzzle. But the queue time was sitting right there in the data the whole time, answering a better question.