Turning 6 Months of Slack Noise Into a Searchable Incident Database

Our CI/CD oncall channels have 4,201 Slack threads going back six months. Two channels: #ci-cd-alerts (2,872 threads, mostly bot-posted failure notifications) and #support-ci-cd (1,329 threads, humans asking for help). Before this week, 66% of the alerts channel was categorized as “uncategorized.” The support channel had zero classification at all.

The categories that did exist were symptom-based. main_ci_blocked. What does that tell you? Main CI is blocked. Thanks. By what? For how long? Is it the same thing that blocked it last Tuesday? No idea. You’d have to open the thread and read it.

I spent a few days building a pipeline to fix this. No LLM calls, no fancy ML. Just keyword matching, GitHub API enrichment, and a bit of structural analysis of Slack reply threads. The result: 0% uncategorized, 970 threads enriched with GitHub Actions run metadata, and 427 auto-generated incident research files.

The pipeline

Three phases. Each one builds on the output of the previous one, but they’re independently re-runnable.

Phase 1: Reclassification

Every thread gets two labels:

L1 tag is about ownership. Who should care about this incident?

• !ourfault — something the DevEx squad broke or misconfigured
• !dependency — an upstream package or external service caused it
• !failed-test — a test is genuinely broken (the test author’s problem)
• !flake — a test that passes on retry (infrastructure or test quality problem)
• — — informational, no action needed

L2 category is about mechanism. What went wrong? 25 categories like acr_dns, test_nccl, runner_offline, gpu_fleet_sssd, image_build_timeout, bazel_drift, and so on.

The classification logic reads the parent message and the first 5 replies, then runs keyword matching. This is the same thing a human oncall person does: you read the bot message, check the replies for “INFRA ERRORS DETECTED” or “Flaky Test,” look at the job names, and form an opinion.

The bot messages in our alert channels have consistent patterns. INFRA ERRORS DETECTED in the reply means infrastructure. A message containing azurecr.io with “connection refused” or “name resolution” means ACR DNS. A reply that says “Flaky Test” with a skip PR link means test flake. These patterns are stable because the bots that produce them are stable.

No LLM needed. The whole thing runs in seconds.

Phase 2: GitHub enrichment

Slack bot messages often contain GitHub Actions run URLs, but just having a URL isn’t enough. You want to know: which job failed? What was the error? When did the run start?

Phase 2 extracts run IDs from thread text via regex, then makes parallel gh api calls to fetch run metadata, failed job names, and error log excerpts. 15 concurrent workers using concurrent.futures. 1,296 unique runs fetched in about 4 minutes, zero errors.

After enrichment, 970 threads have run metadata attached. Of those, 279 have specific failed job names and 181 have error log excerpts. The rest are runs where the failure was in a step that didn’t produce parseable log output, or runs that had already been deleted from GitHub’s retention window.

Phase 3: Research stubs

High-signal incidents (P0/P1 severity, 10+ replies, or recurring across multiple runs) get an auto-generated markdown research file. Each file has a timeline, GitHub run links, failed jobs, error excerpts, and TODO placeholders for root cause, resolution, and lessons learned.

427 files generated. These aren’t postmortems. They’re starting points. The idea is that when a new oncall person encounters an incident that looks familiar, they can search the research files instead of scrolling through Slack. The timeline and error excerpts are already extracted. They just need to fill in the analysis.

What the data looks like now

Before and after:

The L2 category distribution tells a story:

The big number: test_flake_unknown at 61%. That’s the bucket where the Slack bot message says a test flaked, but the message doesn’t include the specific test name or error mechanism. You know something flaked. You don’t know what.

The 61% problem

test_flake_unknown is the honest bucket. It means “I can tell from the Slack message that this was a test flake, but I can’t tell you which test or why it flaked.” The bot messages in #ci-cd-alerts report at the workflow run level, not the individual test level. They say “3 tests failed in shard 2” without listing the test names or error output.

To shrink this bucket, you’d need a different data source entirely. Datadog CI has per-test pass/fail data with error output. Cross-referencing Datadog test-level data with Slack threads would let you sub-classify the flakes into test_nccl, test_timeout, test_oom, test_import_error, etc.

That’s a separate pipeline. The keyword matching approach hits a wall here because the keywords simply aren’t in the Slack messages. You can’t match on text that doesn’t exist.

But 61% unknown is still better than 66% completely uncategorized. The remaining 39% is now fully classified with actionable mechanism labels. And the test_flake_unknown bucket is at least correctly tagged as !flake at L1, so you know it’s a flake even if you don’t know the specifics.

Why this matters for oncall

Oncall for a CI/CD system in a large monorepo is mostly pattern recognition. The same 5-6 failure modes account for the vast majority of alerts. ACR DNS flakes. NCCL timeout on multi-GPU tests. A dependency release breaking an image build. Runners going offline because SSSD lost its Kerberos ticket. Bazel BUILD files drifting from the source tree.

Before the knowledge base, recognizing these patterns required tribal knowledge. You had to have been there when ACR DNS was bouncing last month, or you’d spend an hour investigating something that a veteran oncall person would identify in 30 seconds from the error signature.

Now you can:

1. Filter by L2 category. “Show me all ACR DNS incidents.” Instantly see that there were 15 in the last month, they cluster around Tuesday mornings, and they all resolved within 20 minutes. That’s a pattern. Maybe the geo-replication sync window overlaps with your image push schedule.

2. Read the research files. New to oncall? Instead of reading Slack threads (which are full of tangents, emoji reactions, and “nvm I figured it out”), read the research file. It has the timeline, the error, and (eventually) the resolution.

3. Spot infrastructure investment opportunities. The L2 distribution is basically a priority list. network_timeout at 128 incidents says “your network resilience is costing you 128 person-investigations.” That’s a number you can put in a planning doc.

4. Re-run as data accumulates. The scripts are idempotent. New threads come in, you re-run, the knowledge base grows.

The meta-lesson

You don’t need an LLM for classification when your source data has consistent patterns. Bot-generated Slack messages are, by definition, templated. They use the same phrases, the same structure, the same error formatting every time. That’s exactly the kind of data where keyword matching outperforms everything else: it’s fast, deterministic, debuggable, and costs nothing to run.

The LLM becomes useful at the next layer. Once you’ve classified the incidents mechanically, you can use an LLM to synthesize patterns across incidents (“what do these 15 ACR DNS failures have in common?”), generate draft postmortems, or suggest runbook updates. But the base classification layer should be boring and mechanical.

This is also the kind of infrastructure work that’s invisible until it’s needed. Nobody wakes up and thinks “I wish our oncall knowledge base had better L2 category labels.” They wake up at 2 AM to a pager, grep Slack for the error message, find nothing useful, and spend 45 minutes reinvestigating a problem that was already solved last month. The knowledge base is the thing that turns that 45 minutes into 5.

Scripts

Three Python scripts, all in the oncall workspace:

• classify.py — Phase 1 (reclassification) and Phase 3 (research stub generation). Reads JSONL, applies keyword rules, writes enriched JSONL + markdown stubs.
• enrich.py — Phase 2 (GitHub Actions enrichment). Regex-extracts run IDs from thread text, parallel gh api calls, writes enriched JSONL.
• enrich_research.py — Phase 3b (backfill research stubs for threads that were enriched after initial classification).

All three are re-runnable. They read from and write to the same JSONL files. Classification is deterministic, so re-running with the same data produces the same output. Enrichment skips runs that are already fetched.

The keyword rules are the interesting part. They’re a list of (pattern, L1, L2) tuples, checked in priority order. First match wins. The patterns are simple string containment checks, not regexes. Something like:

rules = [
    ("INFRA ERRORS DETECTED", "!ourfault", "infra_error"),
    ("azurecr.io", "!dependency", "acr_dns"),
    ("Flaky Test", "!flake", "test_flake_unknown"),
    ("NCCL", "!flake", "test_nccl"),
    ("runner is offline", "!ourfault", "runner_offline"),
    # ... 20 more
]

Dead simple. That’s the point.