A Missing Package, a 12-Hour Scramble, and the Art of Workplace Tai Chi

A new library got added to our monorepo. CI passed. The PR merged. Then main broke. A P00 release candidate for a 35B model was blocked on a Friday. Six engineers scrambled for twelve hours. The root cause was two missing lines in two pyproject.toml files.

This post is in two parts. Part 1 is the technical problem: how a hash-based image build system can miss a new package entirely. Part 2 is the more interesting one: how the Slack thread that followed became a masterclass in workplace communication patterns. 职场语言艺术, if you will.

Part 1: The Technical Problem

How image builds work in our monorepo

Our CI builds Docker images for running tests and deployments. These images contain all the Python packages in the monorepo, installed via uv. The workflow goes roughly like this:

1. Hash the contents of uv.lock, root pyproject.toml, and relevant package configs
2. Use that hash as the image tag
3. If an image with that tag already exists in the registry, skip the build
4. If not, build a fresh image

This is a content-addressed system. Change a dependency, the hash changes, a new image gets built. Don’t change anything, the cached image gets reused. Fast and correct, in theory.

The gap: adding a new package

Jessica added a new library called file_content_provider via PR #30362. The code was fine. Tests passed. CI was green. It merged.

Then main broke with:

ModuleNotFoundError: No module named 'file_content_provider'

What happened? Two registration steps were missing:

1. The root pyproject.toml has an [tool.hatch.build.targets.sdist] include list. Every internal package needs to be in that list, or it won’t be included in the source distribution that gets installed into the image.
2. The new package’s own pyproject.toml was missing [tool.hatch.build.targets.wheel] configuration, so even if included, it wouldn’t build correctly.

Neither of these was caught by CI on the PR. Why? Because the PR’s CI ran against the existing image, which was built before the new package existed. The image hash didn’t change in a way that triggered a rebuild. The new package simply didn’t exist in the image, and no test in the PR actually imported it yet.

The breakage showed up later, when downstream code on main tried to import the package.

The rebuild from hell

OK, so just rebuild the image. Simple fix. Except:

Image builds take over an hour. Nadav kicked one off and literally fell asleep on the sofa waiting for it. While he slept, the build failed. Not because of the package issue. Because Ubuntu’s archive.ubuntu.com had a transient outage, and apt-get update failed during the Docker build.

Paweł spent time debugging that. Meanwhile, Dominik found a separate issue with the SGL serving image. Norbert found the actual pyproject.toml fix. David coordinated across threads. Twelve hours from first breakage to green main.

One missing include line. Twelve hours. A P00 release blocked.

The proper fix

Chris had already been working on PR #31876: a pre-commit hook that validates new packages are properly registered in the root pyproject.toml and have correct hatch build config. If this had existed when Jessica’s PR was created, CI would have failed on the PR, not on main.

Norbert asked the right question in the thread: “Why is registering a new package so scary?” It shouldn’t be. But without any validation, it’s easy to forget one of N manual steps, and the failure mode is silent until someone downstream hits it on main.

The pre-commit hook turns a silent deployment-time failure into a loud PR-time failure. That’s the whole fix. Shift left, make the computer check the thing humans forget.

Part 2: The Slack Thread, or, 职场语言艺术

After the incident was mostly resolved, David started a new thread: “Fixing image building in CI as P0?” He tagged engineering leads. This is where things get interesting from a communication perspective.

If you’re an ESL engineer navigating English-language workplaces (hi, that’s me), the technical problem is the easy part. The Slack thread is where you actually need a manual. Let me walk through what happened.

1. The manager’s umbrella response (打太极)

Prashant (DevEx squad manager) replied:

“This is one of the acute pain points of CI. It’s a complex dependency on uv.lock that has evolved over time. While migrating to Bazel is a long term fix (in-progress), @Sriram is actively working on interim improvements to build process with @Zoe @Ryan @Masa”

Notice what this does. The question was pointed: “Should we make this a P0?” A yes or no would create accountability. Instead, Prashant gives a broad answer that distributes responsibility across four people. “Everyone is working on it” means no one specific is on the hook for this particular breakage.

This is a classic manager pattern. It protects individuals from being singled out, which is generous. But it creates ambiguity about who actually owns what. If you’re one of the four people tagged, you’re now in a position where you need to clarify your scope without contradicting your manager.

In Chinese workplace culture, this is textbook 打太极. Redirect the force, don’t absorb it.

2. The scope reframe (撇清关系)

Sriram replied:

“That said, I don’t think this issue is one that will be addressed by my changes (which relate to performance not correctness). I’m still reading through the threads/build logs to understand what happened; once that’s done I’ll have a better sense of what to do.”

This is a very smooth move. Let me unpack it.

Sriram owns the hash-based image tagging system. His PRs (#23781 “Enable pulling images as of some hash”, #22286 “Update interactive tools to use content-based image hash”) are the mechanism that decides whether an image gets rebuilt. The incident was a correctness gap in that mechanism: a new package slipped through without triggering a rebuild.

By labeling his work “performance not correctness,” Sriram reframes himself into the same lane as Zoe and Masa (who work on build speed optimization: persistent volume caching, layer reuse, build time reduction). He’s now on the “making things faster” team, not the “things broke and it’s my system” team.

The second sentence is also well-crafted: “I’m still reading through the threads/build logs” signals diligence and buys time. “Once that’s done I’ll have a better sense” defers commitment without refusing engagement. He’s not saying “not my problem.” He’s saying “I need to understand more before I can say what to do,” which is reasonable and also non-committal.

撇清关系, done elegantly. No fingerprints.

3. The neutral escalation

David’s original message, “Fixing image building in CI as P0?”, deserves credit. It’s framed as a question, not an accusation. He tagged leadership, which forces a response. But the question mark makes it collaborative: “Should we prioritize this?” not “Why didn’t you prevent this?”

This is the best way to escalate in a large org. You’re not blaming anyone. You’re not even asserting that something is broken. You’re just asking whether the priority should change. The implicit message (“this broke main and blocked a P00 release”) is clear to everyone reading, but it’s never stated directly.

4. The “naive question” technique

Norbert, in a different part of the thread:

“Why is registering a new package so scary?”

And later:

“This looks like it could be circumvented with linting.”

Norbert is experienced enough to know the answers. These aren’t real questions. They’re rhetorical moves that highlight the absurdity of the current workflow without criticizing anyone who built it. “Why is this scary?” translates to “This shouldn’t be this hard and the fact that it is means something is wrong.” But by phrasing it as a genuine question, he avoids putting anyone on the defensive.

This technique works especially well in English-language workplaces. A direct statement like “our package registration process is broken” would make the people who built it defensive. A “naive” question invites everyone to agree that yeah, this is weird, and we should fix it.

5. Strategic vulnerability

Nadav:

“Sorry guys.. I fell asleep on the sofa waiting for the image to build.”

This is doing double duty. On the surface, it’s an apology for the delay. Underneath, it’s advocacy: “the image build takes so long that a human can fall asleep waiting for it.” He’s highlighting the severity of the speed problem without writing a formal complaint. The self-deprecating tone makes it impossible to read as aggressive.

Jessica:

“Sorry this caused so much trouble! I didn’t realize adding an unused lib required all this.”

Takes responsibility (graceful) while pointing out the workflow is unreasonable (constructive). “I didn’t realize” is not an excuse. It’s evidence that the process is insufficiently documented and validated. She’s saying “the system failed me” while appearing to say “I failed the system.”

6. How to respond when you’re tagged but it’s not your scope

This is the practical bit. If you’re me (Zoe), Prashant just tagged you alongside three other people for an incident you had nothing to do with. Your actual work is build speed optimization, not the hash-based rebuild correctness that broke. What do you do?

Bad options:

• “That’s Sriram’s system, not mine” (甩锅, throwing someone under the bus)
• Say nothing (looks like you agree it’s your responsibility)
• “We’re all working on it” (perpetuates the ambiguity)

Better approach: describe your specific work concretely, acknowledge the overall pain point, and let the ownership gap speak for itself.

Something like: “On the build speed side, Masa and I are working on [specific thing] to bring image build times from 1hr+ down to [target]. Happy to sync on how that intersects with the correctness improvements.”

What this does:

• Anchors your scope to a specific, valuable track (“build speed side”)
• Names your work concretely so people know what you actually do
• Acknowledges the broader problem (“how that intersects”) without claiming or disclaiming the correctness gap
• Leaves the correctness ownership implicit rather than pointing fingers

You’re drawing a line without drawing a line. 打太极, but the defensive kind.

The meta-lesson

Every person in this thread was doing something reasonable:

• Prashant was protecting his team
• Sriram was managing his scope
• David was doing his job escalating
• Norbert was pushing for systemic fixes
• Nadav and Jessica were being human

Nobody was lying. Nobody was being malicious. But reading the thread as a non-native speaker, you might miss all of it and just see “people discussing a CI incident.” The subtext is where the actual coordination happens.

If you’re learning English workplace communication (我还在学), here’s the cheat sheet:

These aren’t manipulative. They’re how adults coordinate in large organizations where direct confrontation has high social cost. The 职场老油条 (workplace veterans) do this instinctively. The rest of us need to study the tape.