https://songkou.github.io/tags/connectx/Recent content in ConnectX on Council of ElrondHugoenMon, 20 Jul 2026 04:30:00 +0800https://songkou.github.io/posts/rocev2-cisco-cumulus-connectx-end-to-end/Mon, 20 Jul 2026 04:30:00 +0800https://songkou.github.io/posts/rocev2-cisco-cumulus-connectx-end-to-end/<p>A RoCEv2 fabric is not complete when the switch merely forwards UDP port 4791. The endpoints and every switch hop must agree on a traffic-class contract: <strong>DSCP classification, internal priority, egress queue, PFC priority, ECN thresholds, CNP handling, MTU, and DCQCN behavior</strong>.</p> <p>This guide turns that contract into two switch implementations and one host implementation:</p> <ul> <li>Cisco Nexus 9000 with NX-OS Modular QoS CLI (MQC)</li> <li>NVIDIA Spectrum with Cumulus Linux and NVUE</li> <li>Linux servers with NVIDIA ConnectX-5/6/7 adapters</li> </ul> <p>The examples use <strong>DSCP 24 for RoCE data</strong>, <strong>DSCP 48 for CNP</strong>, and <strong>PFC priority 3</strong>. Those values are a design choice, not a protocol requirement. If your environment uses DSCP 26, a different PFC priority, or a vendor profile, change the entire path consistently.</p>