CI Observability: StatsD to Prometheus to Grafana for GitHub Actions

You can’t fix what you can’t measure. That’s the cliché. The more accurate version: you can’t fix what you can’t graph over time. Knowing that today’s build took 40 minutes is useful. Knowing that builds have been trending from 20 minutes to 40 minutes over six weeks — and that the inflection point coincides with a specific Dockerfile change — is how you actually fix things.

GitHub Actions doesn’t give you this. You get per-run logs and a billing page. No time series. No trends. No dashboards. If you want CI observability, you build it yourself.

Here’s how we built it.

The full data flow

GitHub Actions Workflow Completes
  → send_metrics.yml triggered (workflow_run event)
    → Python script queries GitHub API for workflow/job/step timing data
      → Metrics formatted as StatsD gauge/counter strings
        → UDP packet to cicd-statsd-exporter:9125 in K8s
          → statsd-exporter converts to Prometheus metrics
            → Prometheus scrapes statsd-exporter
              → Grafana queries Prometheus
                → Dashboard with pretty lines that go up and to the right
                   (or, more often, up and to the "oh no")

Seven hops from “workflow finished” to “visible on dashboard.” Each one can break independently. Welcome to observability infrastructure.

The trigger: workflow_run

The entry point is a GitHub Actions feature called workflow_run. When any workflow completes — success, failure, cancelled — GitHub can trigger a second workflow in response.

on:
  workflow_run:
    workflows:
      - "Build Yolo Base and Rocket Containers"
      - "Run Tests"
      - "Lint and Type Check"
    types: [completed]

The send_metrics.yml workflow fires after any of these complete. It receives the triggering workflow’s run ID, name, conclusion, and timing metadata. From there, the Python script takes over.

The collector script

The actual metric collection happens in a Python CLI:

uv run python -m ci.cli send-workflow-metrics

This does three things:

1. Query GitHub API for timing data. Using PyGithub, it fetches the workflow run, then iterates through every job, and for interesting jobs, every step. It extracts start times, end times, conclusions, runner labels, and queue durations.

2. Compute derived metrics. Raw start/end times aren’t that useful for dashboards. The script computes:

• duration — how long the job actually ran
• queue_duration — how long the job waited for a runner before starting
• e2e_latency — wall clock time from workflow trigger to job completion (includes queueing)
• time_to_signal — for the overall workflow, how long until you know the result

Queue duration is the sneaky one. A job that takes 5 minutes but waits 12 minutes for a GPU runner has a 17-minute e2e latency. If you only measure duration, you think your CI is fast. If you measure e2e latency, you know it’s not.

3. Format and send as StatsD. Each metric becomes a StatsD gauge line:

yolo_wf_v2.job.duration:342|g|#workflow:build_yolo,job:build_base,runner_type:gpu,status:success

The |g means gauge (point-in-time value). Tags after # provide dimensions for filtering and grouping in Grafana.

Three metric tiers

Not all metrics are equally useful. We collect at three granularities:

Workflow-level: The big picture. “How long does the entire build-and-test cycle take?”

Job-level: Where the time actually goes. “Which job is the bottleneck?”

These are tagged by runner type (cpu, gpu, ubuntu), which lets you see that GPU jobs wait 3x longer for runners than CPU jobs — a useful signal for capacity planning.

Step-level: Maximum detail. “Which step inside this job is slow?”

Step-level is where you catch things like “the docker pull step went from 30 seconds to 17 minutes” or “the pytest step is fine but the artifact upload step takes 4 minutes.” You’d never see this at the job level — it’s buried in the total.

The StatsD handoff

Metrics are sent via UDP to a StatsD exporter running in Kubernetes:

cicd-statsd-exporter.tenant-mai.svc.cluster.local:9125

Why StatsD and not writing directly to Prometheus? Two reasons.

Fire and forget. UDP doesn’t wait for acknowledgment. The CI script sends the metric and moves on. If the exporter is temporarily down, the metric is lost — but the CI job doesn’t hang waiting for a response. For observability data (as opposed to, say, billing data), losing an occasional data point is acceptable. Blocking the CI pipeline because the metrics system is flaky is not.

Protocol simplicity. StatsD’s wire format is a single text line. No Protobuf, no HTTP headers, no authentication tokens. metric_name:value|type|#tags. You can debug it with echo and nc. When your observability pipeline breaks at 2 AM, “simple to debug” is worth more than “technically superior.”

The cicd-statsd-exporter converts StatsD metrics into Prometheus exposition format. Prometheus scrapes it on its normal 15-second interval. From there, it’s standard Prometheus/Grafana — PromQL queries, dashboards, alerts.

Legacy vs modern collector

There are actually two collector scripts in the codebase. This is the kind of thing that happens when a system evolves organically.

collect_workflow_metrics.py — the original. Simpler, uses raw HTTP requests to the GitHub API, minimal metric computation. It sends a basic set of workflow and job durations. No queue duration correction, no step-level metrics.

ci_metrics.py — the current version. Uses PyGithub for API access, computes derived metrics including queue duration and e2e latency, handles edge cases like rerun jobs and cancelled workflows, and includes wait correction — adjusting for the gap between when a workflow is triggered and when GitHub actually starts scheduling its jobs.

The wait correction is subtle but important. GitHub doesn’t start a workflow instantly when you push. There’s a delay — sometimes seconds, sometimes minutes during peak load — between the push event and the first job being queued. Without correction, this delay gets attributed to the first job’s “queue duration,” making it look like the runner pool is slower than it actually is.

Both scripts exist in the codebase. The old one is still importable. No one has deleted it because no one is sure nothing else calls it. This is fine. Everything is fine.

The Chrome Trace bonus

One of the collector’s side outputs is a Chrome Trace file — a JSON format that Chrome’s chrome://tracing viewer understands.

This gives you a Gantt chart of every job in a workflow run. You can see which jobs ran in parallel, which ones were sequential, where the idle gaps are, and exactly how the critical path through your CI pipeline looks.

It’s useful for a different kind of analysis than dashboards. Dashboards show trends over time — “builds are getting slower.” Chrome traces show structure within a single run — “these three jobs could run in parallel but they’re sequential because of an unnecessary needs: dependency.”

A 10-second look at a Chrome trace can reveal “this 20-minute workflow has 8 minutes of idle time between jobs” faster than any amount of Grafana staring.

The cron safety net

The workflow_run trigger handles metric collection for completed workflows. But what if the trigger misses a run? What if send_metrics.yml fails? What if GitHub’s workflow_run event delivery has a hiccup?

A cron job runs every 10 minutes and backfills:

schedule:
  - cron: '*/10 * * * *'

It queries the GitHub API for recent workflow runs, checks which ones have already been reported (using a simple deduplication mechanism), and sends metrics for any that were missed.

Belt and suspenders. The workflow_run trigger handles the common case immediately. The cron handles the edge cases with a 10-minute delay. Between the two, metric coverage is effectively 100%.

What this actually looks like

The Grafana dashboards answer the questions that matter:

“Is CI getting slower?” — Workflow duration time series, 7-day moving average. If the line trends up, something changed. Cross-reference with deployment history to find the cause.

“Why is this specific build slow?” — Job-level breakdown. “Build Yolo Base Image” took 35 minutes (CUDA cache miss), “Upload to Manifold ACR” took 12 minutes (network issue), pytest took 4 minutes (normal). Now you know where to look.

“Are we spending too much time in queue?” — Queue duration by runner type. If GPU queue time averages 8 minutes, you need more GPU runners or you need to reduce GPU job frequency. If CPU queue time is 30 seconds, your CPU runner pool is sized correctly.

“What’s our time to signal?” — The most important metric. From “developer pushes code” to “developer knows if it passed.” If this number is 45 minutes, developers stop waiting for CI and merge on faith. If it’s 10 minutes, they actually look at results.

The dashboard doesn’t fix anything by itself. But it makes the conversation shift from “CI feels slow” to “CI p95 workflow duration increased 40% after commit abc123 on January 15th, which changed the Flash Attention commit pin and invalidated the CUDA layer cache.” One of these statements leads to a fix. The other leads to a meeting.

The meta-lesson

Seven hops is a lot of infrastructure to maintain for “I want to know how long my builds take.” If you’re at a startup with 10 builds a day, this is overkill. Use the GitHub API, dump to a spreadsheet, call it a day.

But at scale — hundreds of builds per day, multiple job types, GPU and CPU runners, distributed across clusters — the spreadsheet approach collapses. You need time series. You need dimensions. You need alerting. You need someone to be able to look at a dashboard and say “GPU queue time spiked at 3 PM because the autoscaler hit its node limit” without manually querying the GitHub API.

The pipeline is seven hops because each hop does one thing well. GitHub provides the raw data. StatsD provides fire-and-forget delivery. Prometheus provides time-series storage and querying. Grafana provides visualization and alerting. Swap any component and the others keep working.

That’s the boring answer. The exciting answer is that once you have the pipeline, you can instrument anything. Cache hit rates. Image sizes over time. Flaky test frequency. Time between merge and deployment. Every metric is just another StatsD line.

The hardest part of CI observability isn’t building the dashboard. It’s building the pipeline that feeds it. Once the pipeline exists, dashboards are free.