NCCL and InfiniBand: The Mental Map Nobody Gave You

You’re working on CI for an ML team. A training job fails with NCCL WARN Recv timed out. Someone in Slack says “check if IB is up.” Another person replies “might be a topo issue, try NCCL_ALGO=Ring.” You nod along and open Google in a new tab.

You know what GPUs are. You know what networking is. But the space where GPUs and networking overlap is this weird, jargon-dense territory where people casually drop terms like “RDMA,” “NVSwitch,” “allreduce,” and “fat-tree topology,” and everyone else seems to know what they mean.

This post is the orientation guide. Not a tutorial. Not a performance tuning manual. Just a map of the territory, so when these terms fly by, you know which layer they live on and how they connect to each other.

Why GPUs need to talk to each other

A single GPU can hold maybe 80GB of memory (an A100) or 192GB (a B200). A modern large language model can be hundreds of gigabytes to multiple terabytes. One GPU cannot hold the whole model, let alone a meaningful batch of training data. So you split the work across many GPUs. Sometimes 8 per node, sometimes thousands across a datacenter.

There are three main ways to split:

Data parallelism is the most common. Think of it like sharding a database: every replica has the full model, but each one processes a different batch of data. After processing, they need to synchronize their gradients so the model stays consistent. This synchronization is a collective operation called allreduce, and it’s where most of the networking action happens.

Model parallelism is what you reach for when the model itself won’t fit on one GPU. You’re slicing the model’s weight matrices across GPUs and recombining partial results mid-computation. This requires extremely fast interconnects because GPUs are blocking on each other mid-forward-pass. Latency here directly becomes training time.

Pipeline parallelism splits the model into stages. GPU 1 runs layers 1-10, GPU 2 runs layers 11-20. Data flows through the pipeline. The communication volume is lower than tensor parallelism, but you get “pipeline bubbles” where GPUs idle while waiting for their input.

In practice, large training runs use all three simultaneously. The point is: GPUs need to exchange data constantly, the data volumes are enormous (gigabytes per second), and the latency requirements are brutal. Regular Ethernet was not designed for this.

The networking stack: four layers

Here’s the map. Everything in GPU networking lives on one of these four layers, and most confusion happens when people talk across layers without realizing it.

Let’s walk through each one.

Physical layer: the wires

Four interconnect technologies matter. They’re not competing alternatives. They solve different problems at different scales.

NVLink: GPU-to-GPU within a node

NVLink is NVIDIA’s proprietary high-speed link between GPUs on the same physical server. Think of it as a private highway between GPUs that bypasses the normal PCIe bus.

Why does it exist? PCIe Gen5 gives you about 64 GB/s per direction. NVLink 4.0 (Hopper generation) gives you 900 GB/s bidirectional. That’s a 14x difference. When two GPUs need to exchange activations during tensor parallelism, PCIe becomes the bottleneck instantly.

NVLink connects GPUs in pairs or groups. Not every GPU can talk to every other GPU over NVLink. On an 8-GPU node like DGX A100, GPUs are connected in a specific topology. Some pairs have direct NVLink connections, others don’t. This topology matters for performance, and it’s why NCCL spends a lot of effort figuring out the optimal communication path.

NVSwitch: all-to-all within a node

NVSwitch is the upgrade from point-to-point NVLink. Instead of a mesh of direct links, NVSwitch provides a crossbar switch that gives every GPU full-bandwidth access to every other GPU on the node. Think of NVLink as point-to-point Ethernet cables versus NVSwitch as a backplane switch.

DGX H100 and DGX B200 use NVSwitch. The result: any GPU on the node can talk to any other GPU at full NVLink speed without routing through intermediate GPUs.

InfiniBand: node-to-node across the cluster

InfiniBand (IB) is the high-speed network fabric that connects servers to each other. If NVLink is the highway between GPUs within a single machine, InfiniBand is the highway system connecting machines across the datacenter.

It’s not Ethernet. It’s a completely separate networking technology with its own cables, switches, protocol stack, and management tools. It was designed from the ground up for low latency and high bandwidth in HPC (High Performance Computing) environments, years before the ML boom.

We’ll go deep on InfiniBand later. For now: it’s the dominant inter-node fabric for GPU clusters.

Ethernet: the fallback

Regular Ethernet (25/100/400 GbE) is what most datacenters already have. You can train across Ethernet. It works. It’s slower and higher-latency than InfiniBand, but for smaller clusters or when the budget doesn’t stretch to IB, it’s fine.

The key innovation for Ethernet in this space is RoCE (RDMA over Converged Ethernet), which bolts RDMA capabilities onto Ethernet hardware. More on RDMA in the protocol section.

When you use which

A single training job often uses all three simultaneously. Tensor parallelism over NVLink within a node, data parallelism over InfiniBand across nodes.

Protocol layer: how data moves

This is the layer where things get unfamiliar if you come from web development. In the web world, you have TCP/IP, and on top of that you have HTTP, and you don’t think much about what the kernel is doing. GPU networking throws all of that out.

RDMA: Remote Direct Memory Access

RDMA is the key concept. Here’s the deal: in normal networking, when application A on machine 1 wants to send data to application B on machine 2, the data travels through the kernel’s network stack. It gets copied from user space to kernel space, processed through TCP/IP layers, handed to the NIC, sent over the wire, received by the other NIC, processed back up through the kernel stack, copied from kernel space to user space. That’s multiple memory copies and CPU involvement at every step.

RDMA eliminates most of that. The NIC reads directly from application A’s memory buffer, sends it over the wire, and the receiving NIC writes directly into application B’s memory buffer. The CPU and kernel are barely involved. No intermediate copies.

“Zero-copy” means the data goes from source buffer to destination buffer without being copied into intermediate buffers along the way. In regular TCP, data gets copied at least 2-3 times between application and wire. RDMA: zero extra copies.

“Kernel bypass” means the application talks directly to the NIC hardware without going through the operating system’s network stack. In web terms, imagine if your Node.js app could write directly to the Ethernet card without going through Linux’s TCP/IP implementation.

Why does this matter? GPU training moves terabytes of gradient data per minute. Every unnecessary memory copy and every microsecond of kernel overhead adds up. At scale, the difference between RDMA and traditional TCP can be 5-10x in latency and significantly higher throughput.

InfiniBand Verbs

“Verbs” is InfiniBand’s API. If TCP has send() and recv(), InfiniBand has verbs. They’re the programming interface for RDMA operations.

The main operations:

• RDMA Write: “I’m going to write data directly into your memory at this address.” The remote side doesn’t even know it happened. No interrupt, no notification unless you ask for one.
• RDMA Read: “I’m going to read data directly from your memory at this address.” Same deal, the remote side is passive.
• Send/Receive: More like traditional message passing. The receiver posts a receive buffer, the sender sends into it.

You’ll hear about Queue Pairs (QPs) and Completion Queues (CQs). Queue Pairs are the communication endpoints (think: sockets). Each QP has a send queue and a receive queue. You post work requests to the send queue, the NIC processes them, and completion events appear in the CQ. It’s an asynchronous model, closer to io_uring than to blocking send()/recv().

RoCE v2: RDMA over Converged Ethernet

RoCE takes the RDMA programming model and runs it over Ethernet hardware instead of InfiniBand. RoCE v2 specifically uses UDP/IP as the transport, so it can be routed across standard IP networks.

The performance gap: InfiniBand native RDMA is faster and lower latency than RoCE, but RoCE is dramatically better than plain TCP. If InfiniBand is a Formula 1 car, RoCE is a very fast sports car, and TCP is a city bus.

RoCE requires some Ethernet switch configuration (PFC, ECN for lossless Ethernet), which makes it more operationally complex than “just plug in Ethernet.” But it doesn’t require a separate InfiniBand fabric.

How this maps to the web world

Collective communication layer: NCCL

Now we’re at the layer that ML engineers actually interact with (indirectly). NCCL (pronounced “nickel”) is the NVIDIA Collective Communications Library.

What NCCL actually does

NCCL implements collective operations. A collective operation is a coordinated communication pattern involving all (or a group of) GPUs. Not point-to-point “send from A to B,” but structured patterns where everyone participates.

The core operations:

AllReduce is king. In standard data-parallel training, after every forward+backward pass, every GPU has computed gradients for its batch. AllReduce averages those gradients across all GPUs, so every GPU ends up with the same averaged gradient. Then they all update their model identically. Without AllReduce, distributed training doesn’t work.

Why NCCL instead of raw MPI?

MPI (Message Passing Interface) has been the standard for distributed HPC communication for decades. It has collective operations too. So why does NCCL exist?

Because MPI was designed for CPUs. NCCL is designed for GPUs.

The difference matters because GPU memory lives on the GPU, not in main RAM. A GPU-aware communication library knows how to:

1. Read directly from GPU memory without copying to CPU first
2. Leverage GPU-specific interconnects like NVLink and NVSwitch
3. Understand GPU topology and route data optimally
4. Use CUDA streams for asynchronous, overlapped communication

MPI can do some of this with extensions (CUDA-aware MPI), but NCCL is purpose-built for it. It also auto-tunes its algorithms based on the detected hardware topology. It knows that GPUs 0 and 1 are connected via NVLink while GPU 0 and GPU 8 are connected via InfiniBand, and it chooses different algorithms accordingly.

NCCL vs MPI vs Gloo

In practice: if you’re training on NVIDIA GPUs, you’re using NCCL. Gloo is the fallback for CPU-only or debugging scenarios. MPI shows up in legacy HPC code and in launchers (mpirun), but the actual GPU communication usually goes through NCCL.

Ring allreduce vs tree allreduce

NCCL implements AllReduce using different algorithms depending on the situation.

Ring allreduce: Arrange N GPUs in a logical ring. Each GPU sends a chunk to its neighbor, which adds it to its own chunk and passes it along. After N-1 steps, every GPU has the full result. The beauty: the bandwidth cost is independent of the number of GPUs. The downside: latency scales linearly with N (each step must complete before the next starts). Good for large messages.

Think of it like passing notes around a circle where each person adds their number before passing it on.

Tree allreduce: Arrange GPUs in a binary tree. Reduce up the tree (leaves send to parents, parents aggregate and send up), then broadcast back down. Latency scales with log(N) instead of N. Better for small messages or high GPU counts where ring latency becomes a problem.

NVLink-aware topologies: NCCL doesn’t just pick ring or tree. It knows the physical topology. If GPUs 0-3 are connected via NVSwitch and GPUs 4-7 are on a different NVSwitch, NCCL will do a local reduction within each NVSwitch domain first, then communicate the partial results across domains. This hierarchical approach minimizes traffic over the slower links.

You generally don’t need to choose the algorithm manually. NCCL auto-selects based on message size, GPU count, and detected topology. But when debugging, knowing these exist helps you understand what NCCL_ALGO=Ring or NCCL_ALGO=Tree is doing.

NCCL environment variables that matter

These are the knobs you’ll see people tuning in job configs, Dockerfiles, and CI scripts:

Common NCCL failure modes

These are the errors you’ll actually see in CI logs:

NCCL WARN Recv timed out or NCCL WARN Send timed out: The most common error. One GPU didn’t hear from another within the timeout. Causes: network partition, a GPU crashed or hung, the other process died, DNS resolution failure, firewall rules blocking traffic.

NCCL WARN Call to ibv_modify_qp failed: InfiniBand Queue Pair setup failed. Usually means IB drivers are not loaded, the IB device doesn’t exist in the container, or the IB subnet manager isn’t running.

NCCL WARN NET/IB : Got completion with error: Data transfer over IB failed. Could be cable issues, switch problems, or the IB link went down mid-transfer.

NCCL WARN Could not find real path of /sys/class/...: NCCL can’t find the GPU’s PCI topology information. Common when running in containers without proper device mounting.

NCCL WARN Bootstrap : no socket interface found: NCCL can’t find a network interface for its bootstrap channel (initial coordination). Set NCCL_SOCKET_IFNAME to the right interface.

Watchdog caught collective operation timeout: This is PyTorch’s wrapper around NCCL. It means dist.barrier() or a collective op didn’t complete within NCCL_TIMEOUT (default 30 minutes). Almost always a symptom, not the root cause. Look at the NCCL logs underneath for the real error.

Debugging flow: When you see a timeout, set NCCL_DEBUG=INFO, reproduce, and look at the initialization logs. NCCL will print the detected topology, chosen algorithms, and network interfaces. The answer is usually in there.

InfiniBand: the deep dive

InfiniBand is a networking technology that predates the ML boom by about 15 years. It was built for HPC clusters (weather simulation, physics, genomics) where latency and bandwidth matter more than cost.

What it is physically

An InfiniBand deployment has three types of hardware:

HCAs (Host Channel Adapters): The network card. This is what sits in the server’s PCIe slot. NVIDIA (formerly Mellanox) ConnectX series cards are the dominant product. A ConnectX-7 card does HDR/NDR InfiniBand. Think of this as the “NIC” but for InfiniBand.

Cables: Either copper (for short runs, up to ~3m) or fiber optic (for longer runs). The connectors are QSFP56 or OSFP depending on the speed generation. They look like chunkier versions of SFP+ Ethernet cables.

Switches: InfiniBand switches connect HCAs together. They look like Ethernet switches but speak InfiniBand protocol. Mellanox/NVIDIA Quantum series switches are the standard. A big cluster might have hundreds of these arranged in a multi-tier topology.

You also need a Subnet Manager (SM), which is software (usually OpenSM or the switch’s built-in SM) that manages the network’s routing tables. InfiniBand doesn’t do routing the way IP does. The subnet manager computes all routes centrally and programs them into the switches. If the SM goes down, existing connections keep working but new ones can’t be established.

The speed generations

InfiniBand speeds have a naming convention that sounds like someone smashing abbreviation keys:

When someone says “we need NDR InfiniBand,” they’re saying “we need 400 Gb/s per port.” The “4x” column is what you typically see because InfiniBand links bundle 4 lanes together.

For reference, 400 Gb/s NDR is roughly on par with 400GbE bandwidth, but with much lower latency (roughly 1 microsecond for IB vs 5-10 microseconds for Ethernet) and native RDMA support.

Key concepts

Queue Pairs (QPs): The fundamental communication abstraction. Every RDMA connection uses a QP. Each QP has a send queue and a receive queue. You post work requests (“send this buffer,” “RDMA-write this data to that address”), and the HCA processes them asynchronously. Think of it as an async I/O submission queue. One process might have hundreds of QPs open simultaneously.

Completion Queues (CQs): Where you learn that a work request finished. You poll the CQ (or get an interrupt) to find out that your RDMA write completed. Similar to how io_uring completion rings work.

Memory Registration: Before you can do RDMA, you must “register” the memory buffers with the HCA. This pins the pages in physical memory (so the OS can’t swap them out) and gives the HCA a mapping from virtual to physical addresses. Memory registration is expensive (takes milliseconds), so it’s done once at setup, not per-transfer.

Protection Domains (PDs): An isolation mechanism. QPs and memory regions belong to a PD. A QP can only access memory registered in its PD. Prevents one process from RDMA-reading another process’s memory.

Partition Keys (P_Keys): Network-level isolation. Like VLANs for InfiniBand. Two nodes can only communicate if they share a P_Key.

Common diagnostic tools

When InfiniBand is misbehaving, these are your first stops:

ibstat: Shows the status of local HCA ports. Is the link up? What speed? What state?

CA: 'mlx5_0'
    Port 1:
        State: Active
        Physical state: LinkUp
        Rate: 200 Gb/sec (4X HDR)

If State isn’t Active or Physical state isn’t LinkUp, you have a physical layer problem (cable, switch port, HCA).

ibstatus: Similar to ibstat but less verbose. Quick “is it up?” check.

ibv_devinfo: Shows detailed HCA capabilities. How many ports, what transports are supported, firmware version. Useful for “do we have the right hardware?” questions.

ibv_rc_pingpong / ib_write_bw / ib_read_lat: Performance testing tools from the perftest package. ib_write_bw measures RDMA write bandwidth between two nodes. ib_read_lat measures latency. If these numbers look bad, the problem is at the IB layer, not in NCCL or your training code.

ibdiagnet: Full fabric diagnostic. Crawls the entire IB network, checks for errors, misconfigurations, cable issues. Takes a while on big fabrics but catches problems that per-node tools miss.

opensm: The Subnet Manager. If routes aren’t being computed, check if OpenSM is running. systemctl status opensm on the management node.

The Mellanox/NVIDIA story

In 2020, NVIDIA acquired Mellanox for $6.9 billion. Mellanox was (and remains, under the NVIDIA brand) the dominant InfiniBand vendor. They make the ConnectX HCA cards, the Quantum switches, and the LinkX cables.

Why did a GPU company buy a networking company? Because GPU training is bottlenecked by communication as much as computation. Owning both the GPUs and the interconnect lets NVIDIA optimize the full stack: GPU memory → NVLink → PCIe → HCA → InfiniBand → HCA → PCIe → NVLink → GPU memory. Technologies like GPUDirect RDMA (the HCA reads/writes GPU memory directly, bypassing CPU memory entirely) only work well when the GPU vendor and NIC vendor cooperate deeply.

The ConnectX card series:

• ConnectX-5: EDR (100 Gb/s). Common in older clusters.
• ConnectX-6: HDR (200 Gb/s). The current workhorse.
• ConnectX-7: NDR (400 Gb/s). Latest generation, shipping with DGX H100/B200.

When someone says “ConnectX-6,” they mean an HDR InfiniBand card that does 200 Gb/s. It’s a specific product, not a protocol version.

How it all connects in a real cluster

Here’s what a typical 8-GPU node looks like internally:

                    ┌─────────────────────────────────────────┐
                    │              GPU Node                    │
                    │                                         │
                    │   GPU0 ═══NVLink═══ GPU1                │
                    │    ║                  ║                  │
                    │   GPU2 ═══NVLink═══ GPU3                │
                    │    ║                  ║                  │
                    │   GPU4 ═══NVLink═══ GPU5                │
                    │    ║                  ║                  │
                    │   GPU6 ═══NVLink═══ GPU7                │
                    │                                         │
                    │   (On NVSwitch nodes, all GPUs connect  │
                    │    to NVSwitch for full all-to-all)      │
                    │                                         │
                    │   CPU0 ───PCIe─── HCA0 (mlx5_0)        │
                    │   CPU1 ───PCIe─── HCA1 (mlx5_1)        │
                    └──────────────┬───────────┬──────────────┘
                                   │           │
                              IB Port 0   IB Port 1
                                   │           │
                    ┌──────────────┴───────────┴──────────────┐
                    │         InfiniBand Leaf Switch           │
                    └──────┬────────┬────────┬────────┬───────┘
                           │        │        │        │
                    ┌──────┴────────┴────────┴────────┴───────┐
                    │         InfiniBand Spine Switch           │
                    └──────┬────────┬────────┬────────┬───────┘
                           │        │        │        │
                    (to other leaf switches → other nodes)

Fat-tree topology

Most large GPU clusters use a fat-tree network topology. If you know web infrastructure, think of it as the datacenter equivalent of a multi-tier load balancer setup.

In a fat-tree, switches are arranged in tiers:

• Leaf switches (Tier 0): Connect directly to nodes. Each node plugs into a leaf switch.
• Spine switches (Tier 1): Connect leaf switches to each other. Every leaf connects to every spine.
• Core switches (Tier 2, in 3-tier fat-trees): Connect spine switches. For very large clusters.

The “fat” part means: as you go up the tree, the aggregate bandwidth stays constant (or close to it). Unlike a regular tree where the root becomes a bottleneck, a fat-tree has enough spine switches to keep up with all the leaves. The bisection bandwidth (total bandwidth across the worst-case cut of the network) equals the total edge bandwidth.

Why does this matter? Because allreduce and allgather generate traffic patterns where every node talks to every other node. A fat-tree ensures that no matter which nodes are communicating, there’s enough bandwidth. A skinnier topology would create hotspots.

Rail-optimized networks

In a rail-optimized (or “rail-only”) network, the InfiniBand fabric is split into independent “rails,” one per HCA per node. If each node has 8 GPUs and 8 HCAs, you build 8 independent smaller InfiniBand networks instead of one big one.

GPU 0 on every node connects to HCA 0, and all HCA 0s connect to the same set of switches (rail 0). GPU 1 connects to HCA 1, on rail 1. And so on.

The trade-off: you lose the ability for any GPU to communicate with any other GPU at full speed (cross-rail traffic has to go through the CPU and PCIe), but you dramatically reduce the number of switch ports needed. For workloads that are structured so that GPU-k on node A mainly talks to GPU-k on node B (which is how NCCL arranges traffic in practice), rail-optimized networks give you nearly the same performance at much lower cost.

The practical mental model

You now have the map. Here’s how to use it.

“I see a NCCL timeout in CI logs. What’s happening?”

Work from the top down:

1. Framework layer: PyTorch’s DistributedDataParallel called a collective op (probably allreduce). It didn’t complete within the timeout.
2. NCCL layer: One or more GPUs didn’t participate. Either a process died, or the network between them is broken. Set NCCL_DEBUG=INFO and look at which rank is stuck.
3. Protocol layer: If NCCL reports IB errors (ibv_modify_qp failed, Got completion with error), the RDMA layer is unhappy. Check IB health.
4. Physical layer: Run ibstat on the affected nodes. Is the link up? At the expected speed? Run ib_write_bw between the nodes to see if the fabric is healthy.

Most NCCL timeouts in CI are caused by: (a) a GPU process segfaulting and the other ranks waiting forever for it, (b) network interface misconfiguration (NCCL_SOCKET_IFNAME pointing to the wrong NIC), or (c) the container environment not exposing IB devices properly.

“We need NDR InfiniBand”

They’re saying: our current network bandwidth is the bottleneck for training performance, and we need 400 Gb/s per port (NDR generation). This implies:

• New ConnectX-7 HCA cards in every node
• New Quantum-2 NDR switches (leaf and spine)
• New cables (OSFP or QSFP112)
• Probably a new fabric design (rail-optimized at NDR speeds)
• Budget: this is a multi-million-dollar ask for a mid-size cluster

“Is it a NCCL problem or a network problem?”

Quick heuristic:

How PyTorch sits on top

When you see this in training code:

import torch.distributed as dist

dist.init_process_group(backend="nccl")
model = DistributedDataParallel(model)

Here’s what happens:

1. init_process_group(backend="nccl") tells PyTorch to use NCCL for collective operations. It establishes a “communicator” across all participating GPUs.
2. NCCL initializes: detects the hardware topology (NVLink? NVSwitch? IB? Ethernet?), establishes connections (QPs if using IB), and picks algorithms.
3. DistributedDataParallel wraps your model. During backward(), it hooks into the gradient computation and automatically calls allreduce to synchronize gradients across GPUs.
4. The allreduce goes through NCCL, which routes it over the appropriate physical links.

The framework layer is thin. PyTorch calls NCCL. NCCL calls IB verbs or NVLink or sockets. The verbs talk to the HCA. The HCA talks to the switch. The interesting engineering is in layers 2 and 3.

The one-page cheat sheet

That’s the map. You don’t need to memorize it. You need to know it exists, so when someone says “the allreduce is bottlenecked on the IB rail,” you can hear: “the gradient synchronization (collective layer) is limited by the InfiniBand network bandwidth (physical layer) on one of the per-GPU network segments (topology).” Three layers in one sentence, and now you know which drawer each word belongs in.