End-to-End RoCEv2 Configuration: Cisco Nexus, Cumulus Linux, and ConnectX

Mon, 20 Jul 2026 04:30:00 +0800

A RoCEv2 fabric is not complete when the switch merely forwards UDP port 4791. The endpoints and every switch hop must agree on a traffic-class contract: DSCP classification, internal priority, egress queue, PFC priority, ECN thresholds, CNP handling, MTU, and DCQCN behavior.

This guide turns that contract into two switch implementations and one host implementation:

Cisco Nexus 9000 with NX-OS Modular QoS CLI (MQC)
NVIDIA Spectrum with Cumulus Linux and NVUE
Linux servers with NVIDIA ConnectX-5/6/7 adapters

The examples use DSCP 24 for RoCE data, DSCP 48 for CNP, and PFC priority 3. Those values are a design choice, not a protocol requirement. If your environment uses DSCP 26, a different PFC priority, or a vendor profile, change the entire path consistently.

Arista_EOS_RoCEv2_Lossless_Config

Mon, 20 Jul 2026 04:00:00 +0800

On Cisco NX-OS, enabling PFC for RoCE normally involves several MQC (Modular QoS CLI class-map -> policy-map -> system qos) layers. On Arista EOS, basic PFC can be enabled directly on an interface with two commands:

priority-flow-control on
priority-flow-control priority 3 no-drop

Those two commands are only the PFC portion of a complete RoCEv2 design. A deployable configuration must also define classification, queue scheduling, ECN, watchdog behavior, buffer thresholds, and the policy on every possible congestion point.

PFC on Council of Elrond

End-to-End RoCEv2 Configuration: Cisco Nexus, Cumulus Linux, and ConnectX

Arista_EOS_RoCEv2_Lossless_Config