NVMe over Fabrics: FC-NVMe, NVMe/TCP, NVMe/RoCE & their optics

NVMe replaced SCSI inside the server because flash is fast enough to expose the protocol overhead; NVMe over Fabrics (NVMe-oF) carries the same command set across a network so that shared storage can keep pace. It runs over three transports that map onto three kinds of network — and therefore three kinds of optics and three sets of physical-layer rules. This page compares them and says what each asks of the links.

Why NVMe-oF

Aspect	SCSI-era (FCP, iSCSI)	NVMe-oF
Command queues	one queue, ~32–256 commands	up to 64k queues × 64k commands; parallel by design
Protocol overhead per I/O	tens of µs of CPU time	a few µs; RDMA transports bypass the CPU
Latency added by the fabric	100–200 µs typical	10–30 µs (RDMA/FC), 30–80 µs (TCP)
Best for	HDD and early flash arrays	all-flash, disaggregated storage, GPU data pipelines

The transports

Transport	Network	Lossless?	Latency	Notes
FC-NVMe (FC-NVMe-2)	Fibre Channel 16/32/64GFC	yes, by credits	very low	runs alongside SCSI FCP on the same fabric and SFPs; zoning and name server unchanged (FC protocol basics)
NVMe/RoCE v2	Ethernet 25/100/200/400G with RDMA NICs	requires PFC/ECN (DCQCN)	very low	RDMA verbs over UDP/IP; needs lossless configuration end to end (Lossless Ethernet)
NVMe/TCP	any Ethernet/IP	no — TCP handles loss	low–medium	no special NICs or switch features; routable; the volume choice for enterprise
NVMe/IB	InfiniBand	yes, by credits	very low	HPC/AI storage (InfiniBand)

All four share the NVMe-oF architecture: a discovery controller tells hosts which subsystems exist; hosts connect queues to I/O controllers; ANA (asymmetric namespace access) handles multipath.

What each asks of the physical layer

Transport	Link requirement	Optics
FC-NVMe	clean FC links — CRC/ITW near zero, credits not exhausted	FC SFPs 16/32/64GFC, vendor-validated (FC optics)
NVMe/RoCE	zero loss: FEC on, PFC on the RoCE class, no CRC errors (each lost frame stalls the RDMA queue with go-back-N retransmission)	25G SFP28 / 100G QSFP28 / 400G; DAC in rack, SR/AOC in row, LR/DR between rooms; FEC per PMD
NVMe/TCP	ordinary; loss costs latency, not correctness	any Ethernet optics
NVMe/IB	as InfiniBand fabrics	IB cables and optics (IB cabling)

RoCE is the sensitive one: a link with a marginal Rx power or a dirty connector produces CRC errors that TCP would absorb silently but that stall RDMA queues visibly. Storage latency spikes that track a port's error counters are the tell (Rx power & link budget, VDM & FEC metrics).

Choosing

Situation	Sensible transport
Existing FC SAN, all-flash arrays	FC-NVMe — same fabric, same optics, enable per target
Greenfield Ethernet storage, mixed vendors, routing between sites	NVMe/TCP
Latency-critical, single-vendor Ethernet, lossless expertise in-house	NVMe/RoCE
HPC/AI cluster with an IB fabric	NVMe/IB or parallel file systems over IB
GPU clusters with Ethernet scale-out	NVMe/RoCE or NVMe/TCP on a separate storage network (GPU fabrics)

Operational notes

Separate networks for storage and compute where possible — or at least separate traffic classes; RoCE needs its own lossless class.
MTU: 9000 on Ethernet transports; consistent end to end.
Multipath: two fabrics or two VLANs/leaves; ANA-aware initiators.
Monitoring: per-port CRC/FEC and pause counters, plus DDM trends — a degrading optic shows up as tail latency long before it fails (Monitoring).
Firmware: NIC/HBA and array firmware compatibility matrices matter as much as optics compatibility (Optics in the SAN).

In CodingBox

The optics of an NVMe-oF fabric are ordinary FC, Ethernet or IB modules; the difference is how little error the transport tolerates. Reading identity and baseline DDM on the bench before installation (Check transceiver, DDM) and coding identities that the array and switch vendors accept (Vendor lock) are the parts of the job that touch the programmer.