The Prototype Layer: Closed PRs and Experimental Infra

I wrote about Zhimin’s GPU test migration playbook here. Six weeks, 77 PRs, five docs, a clean phase-by-phase migration from Condor to Falcon. I studied it from the outside, reverse-engineered the pattern, wrote up the checklist.

Then I searched my own git history and realized I was part of that story. I had just forgotten.

The search

I was looking at Zhimin’s runner configs — the CPU DinD runners, the Falcon node pools, the image caching setup — and something nagged at me. I’d done something with Docker-in-Docker and Falcon before. Weeks ago. I couldn’t remember what, or why, or whether it went anywhere.

So I asked Claude to search for my old PRs. gh search prs --author=shiyuanzheng_microsoft filtered for DinD, Falcon, runner keywords.

Three closed PRs came back. All from early February. All unmerged.

I didn’t remember writing any of them.

The PRs I forgot

In the mango repo (infrastructure):

#9017 — feat(arc-runner): add DinD runner with hostPath cache on cicd node pool Created February 4. Docker-in-Docker runner set targeting Masatoshi’s new cicd node pool. The key detail: a hostPath volume mount so the Docker build cache persists across pods on the same node’s NVMe. Without this, every DinD job starts cold — no layer cache, no reuse, full image pulls every time.

#9073 — feat(arc-runner): add simple CPU runner on cicd node pool for testing Also February 4. I labeled this “Phase 1” — a simple runner without DinD to validate the hardware first. E32ds_v5 SKUs, 32 vCPU, 256GB RAM, maxRunners: 3 for controlled testing. Prove the node pool works before adding complexity.

In the yolo repo (CI workflows):

#16791 — test: switch GPU tests from DinD to K8s runner mode Created February 1. This one was testing whether K8s runner mode (where the kubelet caches images on nodes) would be faster than DinD mode (where every job pulls images from scratch). The problem statement was explicit: DinD mode has no image cache, so every job eats a 17-minute image pull.

All three PRs: closed. Not merged. State: CLOSED.

The timeline I reconstructed

I pieced this together from PR dates, Masatoshi’s PRs, and my own weekly update in the squad channel:

Look at the Feb 6 entry. 52 minutes down to 9 minutes. I wrote that in my own weekly update. And then I forgot about it completely until I searched my git history a month later.

The connection

When I was studying Zhimin’s migration, I noticed things about his setup that seemed like they just existed. Runner configs pointed at node pools. Image builds ran on dedicated hardware. Cache hit rates were high. The infrastructure was just there, ready.

It was ready because someone built it. Specifically, because Masatoshi created the node pools and I prototyped the runner configs and the cache architecture. Our February work was the substrate Zhimin’s March migration ran on.

Here’s how it maps:

Zhimin’s Week 2 lowlight: “No easy way to build our image locally and building CI images is very slow.” That’s the exact problem my hostPath cache experiment addressed. The 17-minute image pull problem that PR #16791 was investigating? That’s the same bottleneck.

Zhimin’s Week 5: CPU DinD runners stood up in Falcon. Those runners ran on the node pools Masatoshi created in early February — the same pools my PRs #9017 and #9073 targeted.

Zhimin’s Week 6: Image build time dropped from 19 minutes to 5 minutes on main CI. The cache architecture that made this possible was the same pattern I proved at 52→9 minutes three weeks earlier.

The 52→9 minute result was a proof-of-concept. It justified Masatoshi’s investment in L-series VMs (better local NVMe for cache). The L-series VMs became the cicd-builder pool. The cicd-builder pool became the foundation for Zhimin’s runner configs. Zhimin’s runners became the platform for the GPU test migration.

Prototype → infrastructure → migration. Three layers, three people, six weeks.

Why the PRs were closed

None of my three PRs merged. They didn’t need to.

They were experiments. I literally labeled them “Phase 1” and “Phase 2.” They proved that hostPath-based Docker cache works on the cicd node pool. They proved that K8s runner mode is faster than DinD for cached image scenarios. They generated the 52→9 minute number.

Then Masatoshi iterated on the infrastructure:

• E32ds_v5 SKUs → L-series VMs (better NVMe for cache)
• My ad-hoc runner configs → dedicated cicd-builder node pool
• Prototype-grade → production-grade

My PRs were superseded. The code was thrown away. The learnings shipped.

This is the thing about prototype work that doesn’t show up in merge stats. Three closed PRs, zero lines of code in production. But the 52→9 minute proof-of-concept justified an infrastructure investment that enabled a six-week migration affecting every GPU team in the org. Closed PRs aren’t failed PRs.

Memory as archaeology

Here’s the part that bothers me.

I could not remember doing this work. It was four weeks ago. I wrote the PRs, paired with Masatoshi, posted the results in my weekly update, and then — nothing. Gone. Overwritten by the next thing and the thing after that.

This is an ADHD thing. Working memory is a sieve. You finish something, context-switch to the next problem, and the previous work evaporates. Not the skills — I still know how Docker caching works. But the episodes. The specific PRs, the timeline, the connection between my work and someone else’s downstream project. All of that was gone until I went looking.

The tools that brought it back:

• gh search prs — found the closed PRs I couldn’t remember
• PR descriptions — past-me documented the rationale (thanks, past-me)
• Weekly updates — the Feb 6 squad update had the 52→9 number
• Masatoshi’s PR timeline — cross-referencing his PRs with mine reconstructed the collaboration

I didn’t remember the story. But the artifacts remembered it for me.

This is why I write weekly updates. This is why PR descriptions matter. This is why you leave a trail even for experiments that don’t ship. Not because you think someone will study your prototype PRs someday — but because you might need to reconstruct what you did, and your brain won’t help you.

The invisible layer

When you read about an impressive migration — six weeks, 77 PRs, five docs, multiple teams onboarded — the narrative centers on the person who did it. And it should. Zhimin’s work was exceptional.

But there’s a layer underneath that you don’t see in the migration timeline. The node pools that already existed. The runner configs that someone already tested. The cache architecture that someone already proved works. The infrastructure investment that someone already justified with a prototype.

That layer doesn’t have its own weekly updates. It doesn’t get its own project tracking. It shows up as closed PRs and a one-line mention in someone else’s weekly update: “Paired with Masa, 52→9 min with cache.”

I’m not saying this to claim credit. I genuinely forgot I did this. I’m saying it because the next time I study someone’s work and think “how did they build all of that in six weeks?” — the answer might include a prototype layer built by someone who has already moved on and forgotten about it.

Including, apparently, me.