Link reliability & monitoring in AI fabrics

A training job is a synchronous computation across thousands of GPUs: every collective operation waits for the slowest link. One flapping transceiver does not slow one server by a fraction — it stalls the whole job, and if it fails outright the job restarts from the last checkpoint. Optics therefore stop being a component and become a reliability budget. This page covers the failure math, the metrics that predict trouble, how fabrics monitor and isolate bad links, and the operational practices that keep failure rates down.

Why one link matters

Event	Effect on a training job
Link flap (seconds)	collectives time out or retry; step time spikes; with NCCL/RCCL a long enough stall aborts the job
Link at reduced speed/width	every all-reduce is paced by it; whole-cluster throughput drops to that link's share
Elevated pre-FEC BER, no uncorrectable	none yet — but the margin is gone; the next warm afternoon it flaps
Uncorrectable FEC errors	packet loss → RDMA retransmission storms → stall
Hard failure	job restart from checkpoint: minutes to an hour of lost compute across the cluster

The cost of a checkpoint restart on a thousand-GPU job is measured in GPU-hours; that is the number to compare with the price of better optics and better QA.

Failure arithmetic

Transceiver reliability is quoted in FIT (failures per 10⁹ device-hours) or MTBF.

Assumption	Value
Optical ends in the fabric	4 000 (a 1 024-GPU two-tier cluster)
FIT per end (mature 400G optics, well cooled)	200–500
Expected hard failures per month	4 000 × 350 × 10⁻⁹ × 720 h ≈ 1
Observed link flap rate in practice	5–20 × higher than hard failures — dirt, marginal margin, thermal, firmware

So hard failures are rare and predictable; flaps and degraded links dominate and are mostly preventable — which is why monitoring and QA pay for themselves.

Metrics that predict problems

Metric	Source	Healthy	Act
Pre-FEC BER per lane	CMIS VDM; host FEC counters	< 10⁻⁷	> 10⁻⁶ rising; > 10⁻⁵ schedule replacement
FEC uncorrectable codewords	host	0	any increment = incident
Symbol errors per lane	host (Ethernet) / `perfquery` (IB)	balanced, near zero	one lane dominating
Rx power per lane vs baseline	DDM	within 1 dB of day one, lanes within 2 dB of each other	−2 dB from baseline
Tx bias trend	DDM	flat	+15–20 % from baseline (Tx bias & ageing)
Module temperature	DDM	< 60 °C	> 65 °C or +10 °C from neighbours
eSNR (CMIS)	VDM	> 18–20 dB	falling on one lane
Link flaps per port	switch logs, fabric manager	0	any
Negotiated speed/width	switch / `ibstat`	full	below rated

Definitions and formats: VDM & FEC metrics, Monitoring.

Monitoring architecture

Fabric	Collection	Tools
InfiniBand	subnet/fabric manager polls every port's counters and cable EEPROM/DDM	UFM-class managers, `ibdiagnet`, `mlxlink -m`
Ethernet (RoCE)	gNMI/OpenConfig streaming of transceiver and FEC state at 10–60 s; SNMP fallback	gnmic → Prometheus/Grafana, vendor telemetry (Reading DDM with tools)
Host side	NIC counters: retransmits, CNPs, out-of-order; NCCL/RCCL logs for timeouts	node exporters
Job side	step-time variance, straggler detection	training framework metrics

Correlate the three: a step-time spike at 14:05, a pre-FEC BER excursion on leaf 7 lane 3 at 14:04 and a module temperature peak in rack 12 is a complete diagnosis.

Isolation and remediation

Automatic port disable / route-around when uncorrectable errors or flaps cross a threshold — fabric managers and NOS support link-error policies; adaptive routing steers flows around a degraded link in the meantime.
Drain the node if its NIC port is the culprit; schedule the job around it.
Bench the module: identity, per-lane DDM against its day-one baseline, checksums, firmware version (Check transceiver, DDM).
Clean and re-inspect the MPO on both sides before blaming the module — dirt is the most common cause (Physical mismatches).
Swap and observe: if the errors follow the module, RMA it; if they stay with the port, look at the cage, the fibre and the far end.

Practices that lower the failure rate

Practice	Effect
Incoming inspection and burn-in (24–72 h under load and temperature)	removes infant mortality before it reaches a job
Baseline per-lane DDM on day one	turns absolute accuracy (±3 dB) into precise deltas
Keep modules < 60 °C — airflow, finned/flat OSFP correctly, no blocked intakes	lifetime doubles per −10 °C
Clean every connector at every mating; caps on unused ports	most flaps are dirt
Firmware currency for modules and hosts	CMIS interop fixes (CMIS issues)
Homogeneous, validated optics per tier	fewer host/module DSP interop surprises
Spares burned-in and baselined, 3–5 %	replacement without a second incident
Trend review weekly	replace ageing lasers on schedule, not on failure

Where the technology is going

Higher lane rates (200G/lane, 1.6T) squeeze margins further; linear (LPO) and co-packaged optics remove the DSP and its monitoring — pushing BER visibility to the host — while promising fewer components and less heat (Modulation & DSP, Optics in AI). Whatever the module, the discipline stays: baseline, monitor deltas, act on trends.

In CodingBox

CodingBox is the bench end of this loop: incoming inspection and burn-in reads, day-one per-lane baselines stored by serial in the code database, and post-mortem reads of pulled modules — identity, firmware, checksums and DDM compared with their own history — so RMA decisions rest on data rather than on the port's last log line.