I wanted something simple: given a CI job name, show me pass/fail rates and duration for the past week. Turns out, figuring out where that data lives was the actual work.
The Starting Point
Our org recently set up Datadog. Someone on the team had a dashboard showing CI pipeline data. Cool — I’ll just query the Datadog API, right?
Not so fast. I had no idea how the data got there. And nobody wrote it down.
So I did what any reasonable person does — grep’d two repos for an hour.
Three Separate Data Pipelines
Here’s what tripped me up: there isn’t one observability system. There are three, and they have nothing to do with each other.
| Data | Where it goes | How |
|---|---|---|
| CI job metrics (pass/fail, duration) | Prometheus via StatsD | Custom Python scripts in GitHub Actions |
| GPU/Infra monitoring | Datadog | Datadog Agent on K8s clusters |
| CI pipeline visibility | Datadog CI Visibility | GitHub App integration (no code) |
The middle row — Datadog Agent — is what I found first. It’s deployed across a bunch of clusters (falcon-dfw-ga, caas-northcentralus, etc.) for GPU monitoring, container logs, network performance. Infrastructure stuff. Not CI.
The first row is a custom pipeline that’s been around longer. GitHub Actions workflows fire on every CI completion, hit the GitHub API for run/job/step data, compute metrics, then ship them as StatsD UDP datagrams to a Prometheus exporter. The code lives in ci/src/ci/workflow_metrics/. It’s actually well built — v2 metrics with full hierarchy, time-to-signal, queue duration, the works.
But the dashboard I saw? Neither of those.
The Actual Answer
The CI data on the Datadog dashboard comes from Datadog CI Visibility — an out-of-the-box integration. You connect it to your GitHub org via a GitHub App in the Datadog UI. No workflow changes, no code, no StatsD. It watches your GitHub Actions runs and ingests everything automatically.
The dashboard URL gave it away:
/dash/integration/297/ci-visibility---pipelines-dashboard
integration/297. Built-in integration dashboard, not custom. The filter was @ci.pipeline.name:"Build Yolo Base and Rocket Containers".
Three systems, three completely different data paths, all showing “CI metrics” in different places. The custom StatsD pipeline feeds Prometheus. The Datadog Agent feeds infra dashboards. CI Visibility feeds the pipeline dashboard — through a GitHub App that lives entirely in the Datadog UI with zero footprint in the codebase.
The Query Part
Once you know it’s CI Visibility, querying is straightforward. Two endpoints:
POST /api/v2/ci/pipelines/analytics/aggregate— aggregated metrics (counts, averages, percentiles)GET /api/v2/ci/pipelines/events— individual run details
The critical filter:
@ci.pipeline.name:"Build Yolo Base and Rocket Containers" @ci.level:pipeline
That @ci.level:pipeline part matters — without it you get job-level and step-level sub-events too, which inflates your counts. Duration comes back in nanoseconds, because of course it does.
The Lesson
The hard part wasn’t writing the query. It was figuring out that three different systems all serve “CI observability” and none of them talk to each other. Grep told me about the StatsD pipeline. Kubernetes manifests told me about the Datadog Agent. But the thing I actually needed — CI Visibility — had zero code footprint. Configured entirely through a web UI.
Sometimes the answer isn’t in the codebase.