InfiniBand fabric essentials

An InfiniBand fabric behaves differently from an Ethernet network in ways that matter when you diagnose a link. This page covers the concepts an optics engineer meets: the subnet manager, lossless flow control, topologies and how a bad link shows up.

The subnet manager

Every IB subnet has one active Subnet Manager (SM). It discovers all ports and switches, assigns local identifiers (LIDs), computes forwarding tables and programs them into every switch. When a link flaps, the SM re-sweeps the fabric — so a marginal optic does not just lose one link, it triggers fabric-wide re-routing events that everyone notices.

Lossless by credits

IB links never drop packets in normal operation: a sender transmits only when the receiver has advertised buffer credits. Congestion therefore back-pressures rather than discards, and a degraded link (rising symbol errors, FEC corrections) slows the whole job that depends on it — the "one slow link" problem in collective operations.

Topologies

Fat-tree (Clos) — the standard: leaf switches connect hosts, spine switches

connect leaves, with enough uplinks to be non-blocking. Most links are leaf–spine optics.

Dragonfly+ and torus variants appear in the largest systems to save switches and

cables.

Rail-optimised designs for AI place each GPU's NIC on a separate "rail" of the

fabric so GPUs with the same rank across servers are one hop apart — see GPU fabric topologies.

RDMA and GPUDirect

Applications use RDMA verbs to move data between node memories without CPU copies; GPUDirect RDMA extends this to GPU memory. The result is that a fabric fault is visible to applications as a stall or a timeout in a collective, often long before an operator sees a down port.

What a bad link looks like

Symptom in the fabric	Physical-layer cause to check
Rising symbol errors / FEC corrected blocks on one port	dirty MPO, marginal Rx power, degrading lane
Link stuck at a lower width or speed	FEC mismatch, module not advertising the expected rate
Port "Not supported", no link	unrecognised module identity — vendor lock
Repeated SM sweeps	a flapping link somewhere — find it by per-port error counters

Follow the usual diagnostic ladder: levels → clean → swap → cable.

In CodingBox

Read a suspect IB module on the bench: per-lane Rx/Tx and bias on the DDM screen show whether one lane of an SR4/DR4 module is dying, and the identity fields show whether the fabric will accept a replacement.