Cabling an AI cluster: counts, distances, media, QA

A GPU cluster is mostly cables. Eight GPUs in a node mean eight 400G (or 800G) scale-out links plus management and storage; a thousand GPUs mean thousands of optical ends across two or three switch tiers, each one a potential training stall. This page turns the fabric designs of GPU fabrics into cable counts, reach classes, connector choices and an installation checklist.

Counting links

For a typical 8-GPU node with one 400G NIC per GPU, rail-optimised two-tier fabric:

Item	Formula	1 024 GPUs (128 nodes)
GPU-to-leaf links	GPUs × 1	1 024 × 400G
Leaf-to-spine links (non-blocking)	= GPU-to-leaf	1 024 × 400G
Optical/cable ends (two per link)	links × 2	4 096
Storage + management NICs	2–4 per node	256–512 more links
Spare optics (3–5 %)	—	150–250

With NDR twin-port switch modules (2 × 400G per OSFP) the switch-side count halves in modules but not in fibres. Three-tier designs add another 1 024 spine-to-core links per 1 024 GPUs. Every one of these ends is a transceiver or cable plug with an identity and DDM (Link reliability & monitoring).

Distances in a rail-optimised layout

Segment	Typical distance	Media
GPU NIC → leaf, same rack	≤ 2 m	passive DAC (400G ≤ 2 m; 800G ≤ 1–1.5 m) — cheapest, zero power, heavy
GPU NIC → leaf, adjacent racks (rail groups span a row)	3–10 m	AEC (≤ 3–5 m) or AOC / SR transceivers over MMF
Leaf → spine, same hall	10–100 m	AOC up to ~50–100 m; SR4/SR8 / VR on OM4 ≤ 50–100 m; DR4/DR8 on SMF 500 m — DR is winning at 400G+
Spine → core / between halls	100 m – 2 km	DR4 / FR4 / 2×FR4 single-mode
Inter-building DCI	2–80 km	LR4, ZR/ZR+ (Coherent & long haul)

Rail-optimised cabling puts GPU i of every node on leaf i, so a node's eight cables fan out to eight different leaves — most GPU-to-leaf links leave the rack, and the DAC share is smaller than in classic ToR designs. Details: Cable internals.

Connector and fibre choices

Optic	Connector	Fibre	Notes
400G DR4 / 800G DR8	MPO-12 APC / MPO-16 APC (or 2 × MPO-12)	SMF, Base-8/Base-16	APC everywhere on single-mode parallel; pinned modules, unpinned patch cords
400G/800G SR8 / VR8	MPO-16 UPC	OM4/OM5	50–100 m; cheaper optics, costlier fibre per metre
400G FR4 / 2×FR4	duplex LC / 2 × LC	SMF	fewer fibres, WDM inside the module
NDR twin-port OSFP	2 × MPO-12 APC	SMF	two logical ports per module; split cables 1:2 / 1:4 (IB cabling)
DAC / AEC	none	twinax	identity on both ends

Polarity method B end to end for MPO trunks; Base-8 trunks for 4-lane optics (Breakout & MPO cabling).

Power and heat of the optics alone

Optic	Power	Per 64-port switch
400G DR4 QSFP-DD	8–12 W	0.5–0.8 kW
800G DR8 / 2×FR4 OSFP	14–18 W	0.9–1.2 kW
400G AOC end	3–5 W	—
800G AEC end	4–6 W	—
DAC	0	—

Switch faceplates therefore need front-to-back airflow without obstruction; finned OSFP in switches, flat-top in NICs; cable bundles must not block intake (Power & thermal, Temperature & voltage). Linear (LPO) optics cut module power roughly in half where the host supports them (Modulation & DSP).

Installation and QA checklist

Incoming inspection of every optic and cable end: identity, checksums, DDM alive, baseline saved (Manufacturing & testing).
Burn-in new optics 24–72 h at operating temperature before deployment.
Inspect and clean every MPO (APC scope tips for APC) at installation; never mate an uninspected connector.
Label both ends: node/GPU/NIC ↔ leaf/port; split cables per leg.
Bend radius and weight management — trays and bundles for DAC/AEC, service loops for fibre.
Bring-up verification: every link at full speed and width, FEC on, per-lane Rx within 2 dB of its siblings, pre-FEC BER < 10⁻⁷ at idle (VDM & FEC metrics).
Record per-lane DDM and identities into the fabric manager / CMDB as the baseline.
Spares by media class and connector; keep them burned-in.

Common installation faults

Fault	Signature
UPC patch on APC DR4	all lanes low by 3–10 dB, both ends
Wrong polarity trunk	all lanes dark
Dirty MPO	one or two lanes low; pre-FEC BER high on those lanes
DAC too long for the host	link at reduced speed or flapping
Split cable leg swapped	ports link to the wrong peers; fabric manager topology mismatch
Finned OSFP in flat cage (or vice versa)	does not seat / overheats

In CodingBox

Incoming inspection at this scale is a workflow, not a spot check: CodingBox reads each module or cable end, verifies identity and checksums, records live per-lane DDM as the baseline and stores it against the serial in the code database, so a module later pulled from a flapping port can be compared with its day-one state.