We had an image that was only built for ARM64. On purpose. “It only runs on Grace Blackwell nodes,” the reasoning went. So why bother with amd64?
Then someone deployed it to an x86_64 CPU-only cluster. Image pull failed. No amd64 variant. Surprise.
What happened
Our monorepo builds a pretraining-experimental image. Masatoshi, who owns the image build pipeline, had been building it single-arch (aarch64 only) because the target hardware was ARM GPU nodes. Reasonable assumption at the time.
Yiran flagged the issue when falcon-phx-ca, a CPU-only x86_64 cluster, tried to pull the image and got nothing. Guangyan and Yazhou jumped in to help mitigate.
Simple fix, right? Just add amd64 to the build matrix.
The second problem
When Masatoshi tried to build the amd64 variant, pip install blew up. The culprit: mai-sgl-kernel-cu131==0.3.20. This package only had an ARM64 wheel published to our private PyPI registry. No amd64 wheel existed. Praveen confirmed one was never built.
So the image fundamentally cannot be built for amd64 today. The multi-arch gap wasn’t just a build config oversight. It was baked into the dependency tree.
Your pyproject.toml can claim cross-platform support all day long. If the actual wheel artifacts only exist for one architecture, the build fails when you try to go multi-arch. Architecture-specific wheels are a hidden landmine that only detonates when you finally try to cross-compile.
The mitigation
Rather than blocking on “get someone to build an amd64 wheel for mai-sgl-kernel,” the team scoped a fast fix: exclude falcon-phx-ca from the experimental image rollout and keep it on the previous stable image. Most experimental image changes are GPU/NVIDIA-specific anyway, so the CPU cluster doesn’t need them urgently.
Masatoshi opened #29635 for the multi-arch build support. Guangyan opened #29643 for the cluster exclusion. (They also discovered they’d been working in parallel on the same problem, which is its own kind of coordination tax.)
Mitigation shipped same day. The full fix, building the missing wheel, is still pending.
The lessons
“This image only runs on X architecture” is an assumption with a shelf life. Infrastructure evolves. Clusters get repurposed. Someone’s staging environment runs on different hardware than production. If you’re building single-arch on purpose, document the assumption and the reason explicitly, so the next person who hits a pull failure knows it’s intentional, not a bug.
Architecture-specific wheels break multi-arch silently. The image builds fine on ARM. All green. You have no signal that amd64 is impossible until you try it. If you depend on private packages, audit whether wheels exist for all your target architectures before you need them.
Mitigation over perfection. The full fix (build the missing wheel, publish it, rebuild the image multi-arch) has multiple teams in the dependency chain. The scoped fix (exclude the cluster that can’t use the image anyway) shipped in hours. Ship the mitigation, then chase the root cause.