CI Observability: Tracing the Training Metrics Pipeline

A teammate pointed me at a Grafana dashboard and said “this is how we track MFU and loss from daily training tests.” I stared at it. Tailscale URL. Prometheus data source. StatsD UDP packets from GPU runners to a cluster exporter. The word he used was 手搓的 — hand-rolled. His word, not mine. But accurate.

I wanted to understand the full pipeline, start to finish. How does a training run’s MFU number end up as a line on a graph? Turns out there are five distinct steps, and each one is doing something slightly clever.

Step 1: the training run writes CSV

It starts with a flag. The daily test jobs pass --csv_writer_metrics=True to their launcher scripts. When this flag is on, the training framework swaps out its normal monitoring backend for a dead-simple CSV writer:

if csv_writer_metrics:
    config.monitoring = registry.get_config(MonitoringSection, "stdout_logging")
    config.monitoring.metrics_logger_config.writer_configs.append(CsvWriterConfig())

This produces metrics_rank_0.csv in the experiment directory. The CSV has everything — train.lm_loss, MFU, Tflops, memory.*, step_time, learning rate, consumed tokens. One row per training step, one column per metric.

Nothing fancy. CSV on disk. The kind of thing you’d write in an afternoon.

Step 2: bash wrappers find the CSV

Two functions in ci_utils.sh handle the extraction. They exist because there are two shapes of experiment output:

extract_metrics_single — for individual experiments. Takes an --exp-name, finds the CSV, calls the Python extractor.

extract_metrics_from_metadata — for batch experiments like perf regression. Reads experiment_metadata.jsonl — a file that lists every run’s name, path, and benchmark config — and loops through each one.

The metadata file is the interesting bit. A single CI job might run five model configurations back-to-back. Each writes its own CSV. The JSONL file is the manifest that ties them together.

Step 3: the Python extractor — this is the real workhorse

extract_training_metrics.py does the heavy lifting. It reads the CSV and converts each metric into a StatsD packet — but not naively. Different metrics get different treatment.

The classification logic:

The smart part is the sampling. Loss metrics get sampled every 50 steps (configurable via --sample-step), with the step number attached as a tag. Timer metrics skip the first N steps to avoid warmup noise. Sliding window metrics — MFU, throughput, memory — get bucketed into windows and emitted as histograms with steps:0-50, steps:50-100 range tags.

This matters because a training run can produce thousands of steps. Emitting a StatsD packet per metric per step would be an absurd amount of UDP traffic. The extractor compresses it down to what’s actually useful for trend analysis.

Step 4: StatsD UDP to a cluster exporter

The metrics get sent as individual UDP packets to:

cicd-statsd-exporter.tenant-mai.svc.cluster.local:9125

A Kubernetes service in the tenant-mai namespace. The packet format is standard StatsD with Datadog-style tags:

yolo_daily_training_metrics_v2.MFU:0.42|g|#experiment:perf_dev_v2_5b,branch_type:main

Tags include the experiment name, workflow name, trigger event, and branch type. For sampled metrics, you also get step:N or steps:X-Y.

Step 5: Prometheus → Grafana

The StatsD exporter forwards to Prometheus. The Grafana dashboard at grafana-las1.tail8d43b.ts.net queries that Prometheus instance. Dashboard name: “MAI Daily Vital Metrics.” Auto-refreshes every 15 minutes.

This is where I expected the story to end. But here’s what caught my attention.

The Datadog gap

I searched Datadog for yolo_daily_training_metrics — the metric prefix used by every training job. Zero results. Nothing. The metrics exist in Prometheus, but they never reach Datadog.

There IS a [W.I.P.] Yolo Tests dashboard in Datadog. Zero widgets. Someone started it and gave up. The empty dashboard is the most telling artifact in this whole pipeline.

Meanwhile, Datadog is where everything else lives — CI pipeline metrics, monitors, alerting, the dashboards the team actually checks daily. The training metrics are on an island. You need Tailscale to reach the Grafana instance. Not everyone has that configured.

Which daily tests feed this pipeline

For reference, here are the jobs that currently emit training metrics:

All of them pass --csv_writer_metrics=True. All of them end up in the same StatsD exporter, the same Prometheus, the same Grafana dashboard.

The case for moving to Datadog

The infrastructure is already 80% there. The StatsD packet format uses Datadog-style tag syntax (#key:value). The tag schema (experiment:, workflow_name:, branch_type:) is exactly what you’d want in Datadog. The only missing piece is routing.

Why bother?

It’s where the team already lives. CI pipeline metrics, monitors, dashboards — all in Datadog. Having training metrics in a separate system means nobody checks them unless something’s already on fire.

Alerting. Datadog monitors can catch MFU regression or loss divergence automatically. Right now, someone has to notice a bad line on the Grafana graph. That someone is usually nobody.

Correlation. Training metrics next to CI pipeline duration and failure rates. Did step time increase because the model changed, or because the runner was throttled? You can answer that in one dashboard if both signals are in the same system.

Access. No Tailscale. No VPN. Everyone on the team has Datadog access already.

How to get there

Three options, in order of how much you’d have to change:

Option 1: Dual-publish from the StatsD exporter. Configure the existing exporter to forward to both Prometheus and a Datadog Agent. Lowest risk — Grafana keeps working, Datadog starts receiving. The StatsD exporter almost certainly supports multiple backends.

Option 2: Direct Datadog API submission. Replace the StatsD UDP calls in the Python extractor with Datadog’s /v2/series API. Cleaner, but you lose the Grafana pipeline unless you maintain both paths.

Option 3: Datadog Agent on GPU runners. Install the DD Agent on the runner nodes. It natively receives StatsD on port 8125. The extractor just points at a different host:port. This is probably the right long-term answer — the Agent also gives you system-level GPU metrics for free.

For any of these, the Datadog dashboard would show:

• MFU trends per experiment — timeseries, compared across runs, with anomaly detection
• Loss curves per model version — step-level granularity, overlaid across nightly runs
• Step time and throughput — catch regressions before researchers notice
• Memory utilization — track creep over time

The empty [W.I.P.] Yolo Tests dashboard is waiting. The pipeline just needs a bridge.