Bazel Migration: Studying a Migration Playbook

2026/03/07

BUILDcidevexinfrakubernetes⭐

I’ve been at the DevEx squad for about six weeks. In that time I’ve been heads-down on Bazel migration and Docker image optimization — my own little corner of the monorepo. But I share a squad channel with Zhimin Xiang, and every Thursday he posts an update about migrating GPU tests from Condor to Falcon.

I started reading those updates the way you read a teammate’s standup — quick scan, thumbs up, move on. Somewhere around week three I started actually studying them. Because what Zhimin was doing wasn’t just infrastructure work. It was a pattern. A repeatable, phase-by-phase approach to large-scale migration that I want to steal for my own projects.

So here’s my attempt to reverse-engineer the playbook.

The migration

Quick context. We run GPU tests across multiple clusters. Condor is the old one. Falcon is the new one — better hardware, different orchestration, different failure modes. Migrating GPU tests from Condor to Falcon means every team that runs GPU CI needs to be onboarded, their tests validated, their workflows updated, and the old path eventually deprecated.

This isn’t a “swap the config and pray” migration. GPU tests have container dependencies, runner environments, image build pipelines, cold-start latency, preemption behavior. Everything is load-bearing. You can’t just flip a switch.

Zhimin did it in six weeks. Here’s how it played out.

Week by week

Week 1 — Lay the foundation

The first thing Zhimin did was not migrate anything.

He optimized the build_image workflow so GPU tests only rebuild the pretrain image when Dockerfiles actually change — saving 8-9 minutes per PR for most runs. He set up logging for CI runners in Falcon so failures would be queryable. He validated the GPU runner environment with Chengji to confirm results matched Condor. He built a manual trigger pipeline so a developer could run Falcon GPU tests through CI on demand.

And he wrote a one-pager: “Setting up CI in a new cluster.”

Key PRs this week: #16421 (image hash-based rebuild), #16457 (image push to Falcon ACR), #16657 (manual Falcon GPU trigger).

No tests migrated. Zero teams onboarded. Just infrastructure, validation, and a doc. If you’re impatient, this looks like a slow start. If you’ve ever tried to migrate something and hit cascading failures on day three because you skipped the validation step — it looks like wisdom.

Week 2 — Prove it works

The build_image optimization landed real numbers: GPU test P90 dropped from ~55 minutes to 44 minutes. That’s not Falcon-specific — that’s an improvement to the whole pipeline. But it builds trust. “The person doing this migration is also making things faster” is a good reputation to have when you’re about to ask teams to change their CI.

Rocket integration tests landed in Falcon (#17832) — first real workload on the new cluster. And immediately, bugs. Runner environment incidents (both self-hosted and GitHub runner pools), uv sync not working on Falcon CPU runners.

This is the phase where you find out what’s actually broken. Zhimin found it by putting real work on the system, not by theorizing.

Week 3 — Document and scale

Here’s where I started paying closer attention. Zhimin wrote two docs this week:

  1. GPU unit test onboarding to Falcon — developer-facing instructions
  2. CI runner reliability plan — root-cause analysis plus improvement roadmap

He also root-caused GPU runner preemption issues and bumped GitHub larger-runner concurrency from 500 to 1500.

The docs matter more than they look. Without the onboarding doc, every team that migrates requires hand-holding. With it, migration becomes self-service. The reliability plan means failures get fixed systematically instead of one-off. This is the week where the work stopped being “Zhimin’s migration” and started being “a migration anyone could participate in.”

Also this week: he tried to deprecate st_tests (#18411, simpletransformer was retired) and had to revert (#18617) because of remaining code deps. And an upgrade to the GH controller broke wait-for-label. Key PRs: #18669 (nightly Falcon GPU tests), #18872 (integrate Falcon into PR and main), #18422 (refactor run_gpu_tests to merge Falcon logic).

Week 4 — Respond to feedback

Developers asked for their Falcon GPU tests to be decoupled from Condor GPU tests — independent execution with annotations for which mode to run. Zhimin implemented it (#20220, #20248) and updated the docs.

This is a detail that’s easy to miss but it’s important. The migration wasn’t a take-it-or-leave-it transition. When users said “we need this to work differently,” the response was “here’s the implementation and updated docs.” Not “that’s not in scope” or “file a ticket.” The migration absorbed the feedback loop.

Also this week: Azure quota approval was slow (it always is), GPU preemption got resolved, and a root-cause on image tag update failures that broke Rocket. Zhimin was sick one day. The work didn’t stop.

Week 5 — Productionize

Three things happened this week that signal the transition from “it works” to “it’s production”:

  1. CPU DinD runners stood up in Falcon — new runner types, not just GPU
  2. Integration test onboarding doc written — extending the playbook beyond unit tests
  3. Cold-start latency design doc — a design document about reducing container init time, shared with HPC and scheduling teams

The integration test doc is the second generation of documentation. Week 3’s doc covered unit tests. Week 5’s doc covers integration tests. The pattern scales by documenting each new capability as it lands.

Also: Rocket GPU test time dropped by ~20 minutes by avoiding redundant image builds (#21370). Performance wins that come from actually operating the system, not from upfront design. Key PRs: #20974 (Launch v2 daily tests on Falcon), #20927 (Rocket eval on CPU DinD runner).

Week 6 — Ship at scale

mai-kernels GPU tests onboarded (#22588 annotation, #23117 PR onboarding, #23556 main CI required) — roughly 10% of all GPU tests — moving through nightly, then PR review, then main CI required workflow. That’s the full ramp. Not “we turned it on and hoped.” Nightly first, then gated PR, then required. Three stages of confidence.

Rocket GPU test image build time: 19 min → 5 min on main CI. 21 min → 13 min on PR-required CI. wait-for-label optimized to skip for CPU-only PRs (#23239), saving unnecessary spin-up across ~1K PRs per week.

Six weeks in, the system is handling real load, multiple teams are onboarded, and there’s a documentation trail that means the next cluster migration won’t start from zero.

The pattern

Okay, so what did I actually learn from watching this? Here’s what I think the phases are.

Phase 1: Instrument before you migrate

Zhimin’s first week was logging, validation, and optimization. He didn’t touch a single team’s tests. He made the platform ready — queryable logs, confirmed runner parity, one manual trigger pipeline for testing.

The instinct is to start migrating immediately. “Let’s get one team over and see what happens.” That works for small migrations. For something touching GPU tests across multiple teams, the first broken test you can’t debug because you don’t have logging will cost you more time than a week of instrumentation would have saved.

The one-pager doc is also phase 1. Write down how to set up CI in a new cluster before you forget the steps. Future-you (or future-teammate) will need this.

Phase 2: Prove with one real workload

Week 2 — Rocket integration tests. One real team, one real workload. Every bug this surfaced (runner incidents, uv sync failures) was a bug that would have hit every subsequent team. Finding them with one team is cheap. Finding them with ten teams is chaos.

This phase is validation through use, not validation through testing. You can write all the test suites you want — they won’t catch “this runner type doesn’t have the right glibc” or “uv sync fails because of a CPU runner image mismatch.” Only real workloads surface real problems.

Phase 3: Document for self-service

Week 3’s onboarding doc is the leverage point. Before the doc, migrating a team requires Zhimin. After the doc, migrating a team requires the team reading instructions.

This is the phase most infra engineers skip. It’s the least fun part. But it’s the difference between a migration that scales linearly with one person’s time and a migration that scales with the number of teams who can read a doc.

The reliability plan is the internal version of this — it’s documentation for yourself, making failure modes legible so you fix them systematically instead of playing whack-a-mole.

Phase 4: Absorb feedback, extend the surface

Week 4’s decoupling feature request and week 5’s integration test onboarding are both extensions. The migration isn’t just “move existing tests.” It’s “make the new platform handle the things people actually need.”

The feedback loop is critical. If your migration plan is rigid — “we’re moving tests from A to B, period” — you’ll lose adoption when teams say “B doesn’t support X.” The successful pattern is to treat feedback as scope, not as resistance.

Phase 5: Ramp through confidence stages

Week 6’s mai-kernels onboarding went nightly → PR review → main CI required. Three stages, each building confidence before the next.

This is not unique to Zhimin — it’s a standard progressive rollout. But it matters that he did it for each team, not just once for the whole migration. Every team’s tests have different failure modes. Nightly catches the environmental issues. PR-gated catches the performance issues. Required catches the reliability issues. Skipping stages saves days and costs weeks.

What was hard

The lowlights across six weeks paint a consistent picture:

None of these were blockers. All of them were friction that required parallel effort. Zhimin was simultaneously building the migration, running oncall, fixing incidents, and responding to developer requests. That’s not a sign of poor planning — it’s a sign that infrastructure work at this scale always has concurrent interrupt-driven work. The playbook has to account for that.

The docs trail

Five documents in six weeks:

  1. “Setting up CI in a new cluster” (Week 1) — Before anything migrates
  2. “GPU unit test onboarding to Falcon” (Week 3) — After validation, before scale
  3. “CI runner reliability plan” (Week 3) — After first production incidents
  4. “Integration test onboarding” (Week 5) — When scope extends beyond unit tests
  5. “Cold-start latency design doc” (Week 5) — When the next bottleneck becomes visible

There’s a rhythm here. Each doc lands at the moment it becomes necessary — not before (speculative docs rot), not after (missing docs become tribal knowledge). The docs aren’t an afterthought. They’re load-bearing infrastructure.

If Zhimin left the squad tomorrow, someone could pick up the migration by reading these five documents. That’s the test.

The playbook

Here’s what I’m taking away. Next time I need to migrate something at this scale — whether it’s Bazel targets, CI pipelines, or anything else — this is the checklist:

Week 1-2: Foundation

Week 2-3: Single-team validation

Week 3-4: Self-service documentation

Week 4-6: Scale and ramp

Throughout:

Wait — I was part of this

Writing this post made me remember something. Around the same time Zhimin was hitting image build bottlenecks in Week 2 — “no easy way to build our image locally and building the image in CI is very slow” — I was hitting the exact same wall from the other side.

Early February. Masatoshi and I were experimenting with faster Docker image builds on Falcon. The problem was DinD mode: cold Docker cache every run, ~17 minute image pulls, no layer reuse between jobs. Every CI run started from scratch. If you’ve ever watched a progress bar crawl through docker pull for the fourth time that day, you know the feeling.

Here’s what we did:

Feb 2-3: Masa created the cicd node pool in the Falcon cluster — dedicated E32ds_v5 nodes with local NVMe storage, separate from the general services pool (mango #8933, #8945, #9028).

Feb 4: I built two runner configurations on top of it:

Feb 1: I also tested switching GPU tests from DinD to K8s runner mode (yolo #16791) — K8s mode uses kubelet image caching instead of DinD, which could eliminate the ~17 min image pull overhead entirely.

The result? With the hostPath cache, pretraining image build time dropped from ~52 minutes to ~9 minutes. That was the proof-of-concept number we needed.

All three of my PRs were closed. Not merged. Closed.

At the time I was confused — did the work not matter? Looking back now, I can see what happened. My PRs were the experiment phase. Masa took the learnings and iterated: E32ds_v5 → L-series VMs with better local NVMe (mango #9987), then a dedicated cicd-builder node pool (mango #10077). Zhimin built his runner configurations and GPU test migration on top of that infrastructure.

The chain looks like this:

My DinD + hostPath experiment (52→9 min proof-of-concept)
    └─→ Masa's L-series node pool (productionized infra)
        └─→ Zhimin's CPU DinD runners on Falcon (Week 5)
            └─→ Zhimin's image build optimization (19→5 min, Week 6)

When Zhimin reports “Rocket GPU test image build time: 19 min → 5 min” — the reason that number was even possible is because the node pool underneath has fast local NVMe storage and persistent Docker cache. That’s the infrastructure Masa and I proved was worth building.

I didn’t know this at the time. I opened two PRs, they got closed, and I moved on to Bazel work. It took writing this blog post — literally tracing Zhimin’s timeline and then going “wait, I was doing something related back then” — for me to see how the pieces fit.

This is the thing about infrastructure work on a team. Your experiment becomes someone else’s foundation. The closed PR isn’t wasted work — it’s the archaeological first layer. But you’ll never know that unless someone traces the timeline. Which, I suppose, is another argument for leaving a trail.

The meta-lesson

I spent my first six weeks on this squad doing deep technical work — Bazel migration, Docker optimization, system torch toolchains. All worthwhile. But watching Zhimin work taught me something about the shape of infrastructure projects that I couldn’t learn from my own work.

The code is the easy part. The hard part is the sequencing — knowing that instrumentation comes before migration, that one-team validation comes before documentation, that documentation comes before scale. Getting the order wrong doesn’t mean you fail. It means you succeed slower, with more pain, and the next person to do it learns nothing from your work because there’s no trail to follow.

Zhimin left a trail. I’m going to leave one too.

Appendix: The PR trail

Zhimin merged 77 PRs in the yolo repo during this period. Here are the milestone ones, grouped by phase:

Week 1 — Foundation:

Week 2 — Prove it works:

Week 3 — Document and scale:

Week 4 — Respond to feedback:

Week 5 — Productionize:

Week 6 — Ship at scale:

The underground layer (my + Masa’s infra experiment):

Where this played out: sq-supercomputing-devex (DevEx squad channel) and support-ci-cd (CI/CD support channel).