NIC ports, speeds and PCIe bandwidth

A NIC's front panel promises "2 × 100G"; whether the server can actually move 200 Gb/s depends on the PCIe slot behind the card, and whether a given module works depends on the cages and the controller's lane map — exactly as on a switch, but with fewer ports and a bus that is easy to under-provision. This page lists the cages and speeds found on NICs, the rules for dual-rate ports and breakout cables, the PCIe arithmetic, the OCP 3.0 form factor, and the power and cooling limits that matter for optics on cards.

Cages and speeds on NICs

Cage compatibility rules are the switch ones (Port types & cages).

Dual-rate ports and speed groups

• Most 25G NICs run each port independently at 10G or 25G — no quad grouping — but a few dual-port controllers share a SerDes group and force the same speed on both ports.
• 100G cards often support 40G modules; 200G/400G cards support 100G modules; going down more than one generation (1G in an SFP28, 10G in a QSFP56 via QSA) is rarely supported.
• Speed is usually auto from the module's compliance codes; forcing (ethtool -s … speed) is needed when the module is dual-rate or unrecognised (Tools).

Breakout on NICs

PCIe: the real port speed

Usable throughput is 85–90 % of raw after encoding and packet overhead. Match ports to lanes:

A 2 × 100G card in a Gen 3 x8 slot links at 100G on both ports and delivers ~60 Gb/s total — the optics are fine, the bus is not. Check lspci -vv for the negotiated link width and speed (LnkSta), and BIOS bifurcation settings.

OCP NIC 3.0

OCP cards sit at the rear of the server with their own airflow channel; module cooling is usually better than on a PCIe card buried between GPUs.

Power and cooling for optics on cards

Multi-host and socket direct

High-end cards can present themselves to two CPU sockets (two x8 or x16 links) or to several hosts. The optical side is unchanged; the split is in PCIe. Relevant for 400G on Gen 4 platforms and for GPU servers where each GPU's NIC must sit on its own PCIe root.

Checklist before buying a NIC for optics

1. Cage type and the speeds the controller supports at that cage.
2. PCIe generation and lanes in the target slot vs total port bandwidth.
3. Breakout needed? Controller must support port split.
4. Module policy of the driver (Optics compatibility on NICs).
5. Cage power budget vs planned modules; auxiliary power connector present.
6. Airflow in the chassis position; consider OCP 3.0 for better cooling.
7. OS/driver support for FEC modes and speed forcing.

In CodingBox

The module's compliance codes and power class tell you whether it can serve the NIC's cage and speed; CodingBox reads them before the module is installed, and the port-split and PCIe questions are settled on the server side (Check transceiver).