When I joined the Microsoft AI team, I sat through my first infrastructure standup and understood roughly 15% of the words being spoken. Someone mentioned “pushing to Falcon ACR and creating the enroot for Condor.” Another person asked if “Manifold staging was pointed at the right Mango endpoint.” A third said “Yoda caught it” like that was a sentence.
I nodded. I took notes. The notes were useless because I didn’t know what any of the nouns meant.
This is the guide I wish someone had handed me on day one. It’s not comprehensive — it’s a decoder ring. Just enough to stop feeling lost when people talk about the infrastructure.
Caveat: This is one DevEx engineer’s mental model. Some of it may be slightly wrong or out of date. If you’re reading this and something’s off, that’s the spirit of the thing — it’s a living document, not a textbook.
The 30-Second Mental Model
Before we get into the specifics, here’s the flow. Burn this into your brain and the rest of the post is just filling in the labels:
You write code in yolo (the monorepo)
↓
CI builds Docker images (GitHub Actions on ARC runners)
↓
Images pushed to multiple ACRs
(inf5acr → Falcon ACR, MangoPDX ACR, Manifold ACR)
↓
Enroot squashfs created from images for HPC clusters
↓
Training jobs run on Condor / Nebius (Slurm + Enroot)
Inference / services run on Manifold (Kubernetes)
↓
Mango manages it all. Yoda watches it all.
Everything below is just explaining what each of those words means.
Compute Clusters: Where the GPUs Live
These are the physical (or cloud) clusters where actual training and inference happen.
| Name | What It Is | What You Need to Know |
|---|---|---|
| Condor | On-prem GPU clusters for training | Multiple clusters: condor1, condor2. Uses Slurm as the job scheduler. You submit jobs with sbatch, check status with squeue. Has shared NFS storage at /mnt/vast/. This is where most large-scale pretraining runs happen. |
| Nebius | Cloud GPU cluster (external provider) | Alternative compute capacity. You’ll see references like prod-eastus-nebius-1. Datasets need explicit Nebius paths defined — they don’t share storage with Condor. |
| Falcon | Azure-based compute infrastructure | Has its own container registry. Builds target both amd64 and arm64 architectures. Used for GPU and CPU workloads. |
When you’ll encounter these: Mostly when someone says “the job failed on Condor” or “we need to test on Nebius too.” As a new hire, you probably won’t be submitting training jobs yourself for a while — but you’ll hear about them constantly.
Infrastructure Platforms: The Control Plane
These are the systems that manage the clusters and services. Think of them as the management layer sitting above the compute.
| Name | What It Is | What You Need to Know |
|---|---|---|
| Mango | Internal infrastructure platform / control plane | Lives at infinity-microsoft/mango. Manages cluster operations, deployments, monitoring. Has its own ACR (MangoPDX). |
| Yoda | Monitoring and alerting system | A component of Mango. Watches cluster health, log analysis, sends notifications. When someone says “Yoda caught it,” they mean the monitoring system detected an issue before a human did. |
| Manifold | Service orchestration / deployment platform | Has its own ACR. Handles multi-arch image distribution. If you hear “Manifold staging” or “Manifold endpoint,” it’s about service deployments (inference, APIs), not training. |
The distinction that confused me the longest: Mango is about infrastructure management (making clusters work). Manifold is about service deployment (running your model as an API). They’re different layers of the stack, but people mention them in the same breath.
Container and Job Runtimes: How Code Actually Runs
This is where it gets a little weird if you come from a pure Kubernetes background.
| Name | What It Is | What You Need to Know |
|---|---|---|
| Slurm | Job scheduler for HPC clusters | Standard in the HPC world, alien if you’re from web/cloud. sbatch submits a job, squeue checks status. Runs on login nodes (slurm-login-0). Think of it as “Kubernetes but for GPU clusters that existed before Kubernetes.” |
| Enroot | Container runtime for Slurm | Converts Docker images into squashfs files that Slurm can run. You’ll see create_enroot_condor in CI — that’s the job building these. If Docker images are shipping containers, enroot squashfs files are the same containers but vacuum-sealed for a different kind of ship. |
| Kubernetes (k8s) | Container orchestration for CI and cloud | Used for GitHub Actions runners (ARC runners) and some cloud workloads. Not used for training — that’s Slurm on Condor. This distinction matters. |
The thing that tripped me up: I kept assuming training ran on Kubernetes because that’s what I knew. It doesn’t. Training runs on Slurm+Enroot on dedicated GPU clusters. Kubernetes handles CI runners and service deployments. Two separate worlds.
CI/CD Infrastructure: How Code Gets Built and Shipped
| Name | What It Is | What You Need to Know |
|---|---|---|
| ARC Runners | Self-hosted GitHub Actions runners on k8s | Actions Runner Controller. Managed in the mango repo. The runner names encode their specs — ubuntu-2404-x64-32c.128g.1200g means Ubuntu 24.04, x64, 32 cores, 128GB RAM, 1200GB disk. |
| Rocket | Optimized build/image variant | “Rocket containers” are optimized images with different build parameters. Has its own enroot variants (create_rocket_enroot_condor). You’ll see “rocket builds” in CI pipelines. |
| Yolo | The monorepo | infinity-microsoft/yolo. Contains mai_config, mai_layers, mai_trainer, mai_distributed, and ~100 other packages. The CI pipeline “Build Yolo Base and Rocket Containers” builds Docker images, pushes to all ACRs, and creates enroot images. Currently migrating to Bazel (that’s my job). |
The ACR Zoo: Why There Are So Many Container Registries
This one confused me for weeks. “Why do we have four container registries?” The answer: different infrastructure targets need images in different places, and network topology means you can’t just pull from one central location.
| Registry | What It’s For |
|---|---|
| inf5acr | Primary build target. Where yolo base images get built first. |
| Falcon ACR | Falcon infrastructure. Gets both amd64 and arm64 builds. |
| MangoPDX ACR | Mango/infra images. The platform control plane’s own registry. |
| Manifold ACR | Manifold deployment images. Multi-arch manifests live here. |
The CI pipeline builds once (to inf5acr), then distributes to the others. When an image build fails, knowing which ACR it failed on tells you a lot about what went wrong — DNS resolution, auth, network path, or the build itself.
Slack Channels: Where to Find Things
You will be added to many channels. These are the ones that actually matter for understanding what’s happening:
| Channel | What You’ll Find |
|---|---|
#sq-supercomputing-devex |
Our squad — CI, build systems, Bazel, runners |
#ws-ai-infra-updates |
Weekly infra updates (Yoda, Mango, cluster status) |
#sq-ai-infra-training |
Training infrastructure issues, dataset paths, cluster problems |
#support-yolo |
Yolo monorepo support — “why is my build failing” goes here |
#msi-coreai-manifold |
Manifold platform discussions |
#yoda-notifications |
Automated cluster monitoring alerts. High volume. I filter these. |
Pro tip: Star #ws-ai-infra-updates and read it every Monday. It’s the fastest way to build context on what the infrastructure team is working on and what’s broken.
The Glossary You’ll Wish You Had in Meetings
Some terms that come up in conversation and aren’t immediately obvious:
| Term | What People Mean |
|---|---|
| “The bake” | docker buildx bake — multi-target Docker build. “The bake failed” = image build failed. |
| “Push to Falcon” | Distribute a container image to Falcon ACR. |
| “Enroot it” | Convert a Docker image to squashfs for Slurm clusters. |
| “On Condor” | Running on the on-prem GPU cluster. |
| “Manifold endpoint” | A deployed service URL. Usually a model serving inference. |
| “ARC runner” | A GitHub Actions self-hosted runner. Not related to Azure ARC (that’s a different thing). |
| “Yoda alert” | Automated monitoring notification from the Mango platform. |
“The vast” / /mnt/vast/ |
Shared NFS storage on Condor clusters. Where datasets and checkpoints live. |
| “Rocket build” | Optimized container variant. Faster to build, different optimization flags. |
| “Base image” | The yolo Docker image with system packages, CUDA libs, Python deps. Everything your code needs to run. |
What I’m Still Figuring Out
Honestly? Some of the boundaries between these systems are still fuzzy to me. I’m not 100% sure where Mango ends and Manifold begins in every case. I don’t fully understand the Nebius networking setup. And I still have to look up the runner spec format every time.
That’s fine. You don’t need to understand everything to be effective. You need to understand the flow (code → image → registry → cluster) and know which noun means which layer. The rest you pick up by breaking things and asking questions in #support-yolo.
Welcome to the team. It gets less confusing. Mostly.