CI Observability: Six Layers Deep in a Grafana Panel

I wanted to know where one number came from.

We have a Grafana dashboard for GPU job queuing — Kueue metrics, pending workloads, admitted workloads, resource usage per cluster queue. Standard stuff. I clicked on one panel showing kueue_pending_workloads across clusters and thought: where does this number actually come from? Like, what is the physical chain of events between “a GPU job is waiting” and “a line goes up on my screen”?

Six layers. The answer is six layers.

Layer 1: The source

Kueue is a Kubernetes-native job queuing system. It manages which workloads get admitted to run based on cluster capacity, quotas, and priorities. The controller-manager pod runs in the kueue-system namespace, and like any well-behaved Kubernetes controller, it exposes a /metrics endpoint — port 8443, HTTPS.

The metrics are what you’d expect from a queue manager:

• kueue_pending_workloads — jobs waiting for resources
• kueue_admitted_active_workloads — jobs currently running
• kueue_cluster_queue_resource_usage — how much of each resource quota is consumed

Standard Prometheus format. Counter, gauge, histogram. Nothing exotic. This is the easy part.

So who’s reading this endpoint?

Layer 2: The contract

A ServiceMonitor CRD tells the metrics collector what to scrape. It’s a declarative config — “hey, there’s an app in this namespace, on this port, at this path, and I want you to scrape it every 60 seconds.”

Here’s the first wrinkle. The ServiceMonitor CRD isn’t the standard Prometheus Operator monitoring.coreos.com/v1. It’s Azure’s flavor: azmonitoring.coreos.com/v1. Same shape, different API group. Kustomize patches the API version transparently at deploy time, so the upstream Kueue Helm chart doesn’t need to know about this. You’d never notice unless you looked at the applied manifest instead of the source chart.

The ServiceMonitor also has metricRelabelings that drop all go_* runtime metrics. Smart — those are Go runtime stats (GC pause times, goroutine counts) that the Kueue controller emits by default but nobody’s dashboarding. Dropping them at collection time saves ingestion cost.

Layer 3: The collector

Azure Monitor Agent (AMA) runs in kube-system with a Prometheus target allocator sidecar. The target allocator watches for ServiceMonitor CRDs (the Azure-flavored ones), discovers which pods match, and distributes scrape targets across AMA replicas.

Every 60 seconds, AMA hits the Kueue controller’s /metrics endpoint, parses the Prometheus text format, and buffers the samples. So far this is conceptually identical to running a vanilla Prometheus server — just with Microsoft’s agent doing the scraping instead.

The difference is what happens next.

Layer 4: The transport

In a self-hosted Prometheus setup, scraped metrics get written to local disk. Done. Here, they go through Azure’s ingestion pipeline.

Data Collection Rules (DCR) define where metrics should go. They specify the stream name (Microsoft-PrometheusMetrics), any transformations, and which Data Collection Endpoint (DCE) to forward through. The DCE is the actual network endpoint that receives the data — and it’s a private endpoint. No public internet. The AMA pod talks to the DCE through the cluster’s private network.

This is the layer where “scrape some metrics” becomes “transport data through a managed cloud pipeline.” If you’ve ever wondered why cloud observability costs what it does — this is it. You’re paying for a dedicated private ingestion path per cluster.

Layer 5: The store

Metrics land in an Azure Monitor Workspace (AMW). One per cluster. Not shared, not federated — each cluster gets its own isolated Prometheus-compatible store. The backend is Geneva Metrics, which is Microsoft’s internal metrics system. AMW wraps it with a Prometheus-compatible query API so Grafana can talk to it using PromQL.

One AMW per cluster means there’s no cross-cluster query. You can’t write kueue_pending_workloads{cluster="east"} from a single datasource. The data is physically separated.

This explains something I’d always been mildly confused by.

Layer 6: The dashboard

Azure Managed Grafana connects to each AMW through Managed Private Endpoints. One Prometheus datasource per AMW. The var-cluster dropdown in the dashboard? It’s not filtering data within a single Prometheus instance. It’s literally switching between entirely separate backends. Choose “cluster-east” and Grafana queries AMW-east. Choose “cluster-west” and it queries AMW-west.

That dropdown is doing a lot more than it looks like.

The full picture

Here’s the chain for one metric sample:

Kueue controller pod
  → /metrics endpoint (port 8443, HTTPS)
    → ServiceMonitor CRD (Azure-flavored, Kustomize-patched)
      → Azure Monitor Agent + target allocator (60s scrape)
        → Data Collection Rule → Data Collection Endpoint (private)
          → Azure Monitor Workspace (per-cluster, Geneva backend)
            → Azure Managed Grafana (private endpoint, PromQL)
              → line goes up on your screen

Six hops. One of them is a CRD API group swap. One is a managed private network tunnel. One is Microsoft’s internal metrics backend wearing a Prometheus costume. And at the end of it all, a line chart says “3 pending workloads.”

Bonus discoveries

While tracing this, I found two things I wasn’t expecting:

The Kueue version is a fork. Not upstream Kueue — a custom build with additional CRD support. The images come from an internal registry. This matters if you ever try to debug Kueue behavior by reading the upstream docs. The behavior might not match.

Newer clusters have a dual metrics path. In addition to the AMW pipeline, some clusters also send metrics to Azure Data Explorer via pod annotations. Same data, different destination — ADX for long-term Kusto queries, AMW for real-time Grafana dashboards. Two parallel observability pipelines, sourced from the same /metrics endpoint.

Why bother tracing this

I spent a couple hours on this. Arguably, I didn’t need to. The dashboard works. The numbers are correct. Nobody asked me to understand the pipeline.

But now I know:

• Where to look when metrics go missing. Is it the ServiceMonitor selector? The AMA agent? The DCR? The private endpoint? There are six places this can break, and they each fail differently.
• Why the cluster dropdown behaves the way it does. It’s not a label filter. It’s a datasource switch. If AMW-east is down, that dropdown option returns nothing — the other clusters are fine.
• What we’re actually paying for. Private endpoints, per-cluster AMWs, managed Grafana with private connectivity. Every layer in this stack has a cost line item.
• Where the data duplication is. Some clusters are sending the same metrics to two backends. That’s intentional, but it’s also double the ingestion cost.

Most engineers interact with observability at the edges: they write code that emits metrics, and they read dashboards. The six layers in between are someone else’s problem — until they’re not. Until a dashboard panel goes blank during an incident and you need to figure out which of six links in the chain broke.

One click on a Grafana panel. Six layers of infrastructure. That’s modern observability.