Lossless Ethernet for storage: DCB, PFC, ECN

Ethernet drops frames when it is congested; storage protocols built for Fibre Channel or RDMA assume nothing is ever dropped. Data Center Bridging (DCB) closes the gap with per-priority flow control and bandwidth allocation, and RoCE adds congestion signalling on top. The result is only as lossless as its worst link: a marginal optic that drops one frame in a million is a storage incident. This page explains the mechanisms and translates them into physical-layer requirements.

The DCB toolkit

Standard	Name	What it does
IEEE 802.1Qbb	PFC — priority-based flow control	pause frames per traffic class (0–7): a congested receiver stops one class without stopping the port
IEEE 802.1Qaz	ETS — enhanced transmission selection	guaranteed bandwidth shares per class (e.g. 50 % storage, 50 % LAN)
IEEE 802.1Qaz	DCBX — DCB exchange	LLDP-based negotiation of PFC/ETS settings between switch and NIC
IEEE 802.1Qau	CN — congestion notification	rarely deployed
RFC 3168 + DCQCN	ECN marking + rate control	the switch marks packets when queues build; RoCE NICs slow the flow before PFC is needed

How RoCE stays lossless

Storage/RDMA traffic is tagged with a dedicated priority (often 3 or 4) via DSCP or 802.1p.
PFC is enabled only for that priority on every port along the path, with buffer headroom sized for the link speed and cable length (in-flight bytes).
ECN thresholds mark packets early; NICs run DCQCN to reduce their rate, so PFC fires only as a last resort.
ETS guarantees the storage class its bandwidth under LAN load.
DCBX keeps NIC and switch settings consistent; mismatches silently break losslessness.

FC and InfiniBand achieve the same with credits at every link by default; Ethernet needs it configured, verified and monitored (FC protocol basics, InfiniBand transport).

What can go wrong

Problem	Effect	Detection
PFC not enabled on one hop / wrong priority	drops under load; RDMA retransmits; latency spikes	pause counters zero where they should not be; drops on the class
PFC storm	a stuck receiver pauses upstream indefinitely; congestion spreads fabric-wide	pause frames flooding; watchdog counters
PFC deadlock	cyclic buffer dependency in Clos fabrics; all traffic on the class stops	requires deadlock detection / watchdog to break
Headroom too small for long cables/optics	drops despite PFC	drops with pause counters active
ECN thresholds wrong	too early: throughput loss; too late: PFC floods	ECN mark counters, CNP counters on NICs
Physical errors on a link (CRC, FEC uncorrectable)	frames lost that no flow control can recover	port error counters, FEC counters — VDM & FEC metrics

iSCSI without DCB

iSCSI runs over TCP and survives loss, but pays for it in latency and retransmission. Best practice regardless of DCB:

dedicated VLANs/subnets per fabric, two fabrics, MPIO on hosts;
jumbo frames (MTU 9000) end to end;
no oversubscription between initiators and targets; 25G/100G uplinks sized for the array;
FEC on for every 25G+ link; CRC counters at zero;
optionally PFC on the iSCSI class to smooth micro-bursts (many arrays recommend it).

Physical-layer requirements

Requirement	Why	Practice
FEC enabled and matched on every 25G+ link	RoCE cannot tolerate the raw BER of unprotected 25G/50G lanes	FEC & link training
CRC and uncorrectable FEC = 0	each lost frame stalls a queue	alert on any increment
Rx power inside the window with margin	marginal optics produce bursts of errors under temperature	Rx power & link budget
Consistent speed/duplex/AN	DAC needs AN + link training both ends	Copper & DAC
Cable length within headroom assumptions	headroom is computed from in-flight bytes	prefer DAC in rack, SR/AOC in row, DR/LR between rooms
Identity accepted by NIC and switch	rejected optics fall back to no link or reduced speed	Vendor lock

Monitoring checklist

Per port: pause frames sent/received per priority, ECN marks, drops per class, CRC, FEC corrected/uncorrectable, link flaps.
Per NIC: CNPs sent/received, retransmissions, out-of-sequence, timeouts.
Per optic: Rx/Tx power and temperature trends (Monitoring).
Correlate storage tail-latency spikes with the above — the physical layer is guilty more often than the array.

In CodingBox

A lossless fabric is built from ordinary optics that are simply held to a higher standard. Incoming inspection on the bench — identity, checksums, live DDM against typical values, baseline stored — is how marginal modules are kept out of a fabric where one bad link costs an application its latency SLA (Check transceiver, DDM).