RDMA Performance Tuning: From Fresh Install to Line-Rate Bandwidth

The number-one complaint with RDMA is simple: the bandwidth won’t fill up. A 100G card pushing only 60G, two days spent checking drivers, optics, and the peer’s configuration — and in the end the fix is a single PCIe parameter.

When RDMA underperforms, 90% of the time the cause is not exotic. It’s some factory default that was never touched: the switch buffer carved too small, the PCIe read request size left at its tiny default, the MTU misaligned on one segment of the path. None of these is hard to fix on its own — the hard part is knowing which one to check.

This post is a complete path from “card just arrived” to “bandwidth at line rate”: seven checks in order, a one-shot tuning script, and a fault-isolation table to come back to the next time RDMA is slow. A second part covers the other classic complaint — packet loss — with a five-step triage ordered by how often each cause is actually the culprit; the first three steps resolve about 70% of cases.

Applies to: ConnectX-5/6/7 · MLNX_OFED · RoCEv2 / InfiniBand

This is research from internet, could not verify in lab due to hardware limitation. May not be accurate.

01 — Confirm the hardware

Rule out the physical layer first.

Is the card even recognized?

ibstat

PCIe link state — this is the one that matters most:

lspci -vv -s <BDF> | grep -E "Width|Speed"

You must see:

If the output contains downgraded, the link is degraded and bandwidth will never reach line rate. Common causes: the card isn’t fully seated, the BIOS has the slot pinned to Gen3, or the slot shares lanes with a GPU.

NUMA alignment:

ibv_devinfo -d mlx5_0 | grep numa

NIC on NUMA node 1 while the application runs on NUMA node 0? Cross-NUMA latency will hurt — a lot. This trap is especially easy to step into on dual-socket servers. (And it isn’t about inter-socket bandwidth — it’s the extra hop cost per operation. At high rates, every extra step is wasted performance.)

Max out the RX/TX queues:

ethtool -l <iface>
ethtool -L <iface> combined <max>

02 — PCIe MaxReadReq

The factory default is 512 bytes — far too small for large RDMA transfers.

# Check the current value
lspci -vv -s <BDF> | grep -i MaxRead

# Set it to 4096 (value 5 = 4096)
setpci -s <BDF> 68.w=5

Large-message bandwidth typically improves 10–20% after this change.

⚠️ The setting is lost on reboot. Add it to rc.local or wrap it in a systemd service.

03 — CPU and interrupt binding

cpupower frequency-set -g performance
systemctl stop irqbalance
set_irq_affinity_cpulist.sh 0-7 <iface>

You can skip this — but RDMA latency will then spike intermittently, and you’ll go crazy trying to find out why.

04 — mlnx_tune

NVIDIA’s official tuning tool, with preset profiles per scenario:

mlnx_tune -p HIGH_THROUGHPUT

It automatically tunes receive queue depth, interrupt coalescing, and ACK delay.

05 — Queue parallelism

A single QP has a bandwidth ceiling. The perftest defaults use only one QP, which cannot fill the pipe.

Baseline run:

ib_write_bw -d mlx5_0 --report_gbits -F -D 10

Multi-QP with large messages — the real performance:

ib_write_bw -d mlx5_0 -q 16 -t 128 -s 1000000 -n 100000 --report_gbits

Parameter reference:

⚠️ Two pitfalls:

1. On CX-6 DX, running more than 8 QPs causes MR cache thrashing and eats about 22% of throughput. Add --mr_per_qp.
2. On CX-7, don’t let the total queue count across VFs exceed 32. Measured: 2 VF × 16 queues = 88.4 Mpps; 4 VF × 16 queues = 51.3 Mpps — cut in half.

What to expect from a 100G NIC:

Measured on DGX Spark + CX-7: ib_send_bw -s 1048576 -q 8 -r 1024 -Q 64 → 93–95 Gbps.

06 — MTU

The highest-hit-rate trap on the network side. The switches are configured for jumbo frames, but one Leaf-to-Spine uplink was forgotten. One wrong segment ruins the whole path.

End-to-end verification — do not skip this:

ping -M do -s 8972 <peer_IP>

If it fails, check segment by segment: Leaf downlink, uplink, Spine port, VLAN interface. Missing any one of them breaks it.

Configuration:

# Host side
ip link set <iface> mtu 4200

# Switch side (Nexus example)
interface ethernet 1/1
  mtu 9216

⚠️ Bigger is not always better. Byte-jitter testing showed some NICs perform worse at 4KB MTU than 1KB under high connection counts. When a new card arrives, run ib_write_bw at 1KB / 2KB / 4KB and keep whichever wins.

07 — ECN + PFC validation

Check the network side if bandwidth still won’t climb after going through the first six steps.

# Switch: PFC trigger counters
show interface priority-flow-control

# NIC: ECN marks
ethtool -S <iface> | grep ecn

⚠️ PFC counters continuously climbing = something is misconfigured. On a healthy link PFC almost never fires.

Field-tested values, ready to use:

⚠️ Never let PFC and TCP share the same physical port. TCP retransmissions and PFC pause frames stack up and can deadlock PFC. Put RDMA on one port and TCP on another — physical isolation.

The tuning script

New machine comes online — run this:

#!/bin/bash
# Applies to: ConnectX-5/6/7 + MLNX_OFED v24+

# Interface to tune: first argument, or ens1np0 if none given
IFACE=${1:-ens1np0}
# Resolve the NIC's PCIe address (BDF) from the interface name
BDF=$(ethtool -i $IFACE | grep bus-info | awk '{print $2}')

# Pin all CPU cores at max frequency — no downclocking mid-transfer
cpupower frequency-set -g performance 2>/dev/null
# Stop the IRQ balancer so interrupts stop migrating between cores
systemctl stop irqbalance 2>/dev/null
# Pin the NIC's interrupts to cores 0-7 (same NUMA node as the NIC)
set_irq_affinity_cpulist.sh 0-7 $IFACE
# Jumbo frames — must match the switch config end to end
ip link set $IFACE mtu 4200
# PCIe MaxReadReq 512 -> 4096 (register 68, value 5 = 4096)
setpci -s $BDF 68.w=5
# Max out the RX/TX ring buffers to absorb bursts
ethtool -G $IFACE rx 8192 tx 8192
# Apply NVIDIA's throughput profile (queue depth, coalescing, ACK delay)
mlnx_tune -p HIGH_THROUGHPUT

echo "Done. Verify with:"
echo "ib_write_bw -d mlx5_0 -q 16 -t 128 --report_gbits -F -D 10"

Fault-isolation table

Part 2 — Packet loss: check the easy things first

A war story that sets the rule: on a freshly inherited cluster, RDMA bandwidth was stuck at half of what ib_write_bw should show. First suspicion was cabling — half a day spent reseating, swapping cables, and measuring optical power. The actual cause: one port on the peer switch still at the default MTU of 1500.

The habit that came out of it: when RDMA drops packets, start with what’s easiest to verify, not with what looks most likely. The order below is by real-world frequency — how often each step turns out to be the answer — not by textbook OSI layers. Knock out the 5-minute checks before going deep.

The 10-second precheck. Same hardware check as step 01 of Part 1 — ibstat and lspci -vv -s <BDF> | grep -E "Width|Speed". If the PCIe link shows downgraded, bandwidth is capped no matter what else you fix. It costs 10 seconds; always do it first.

Step 1 — Cables and optics (~20% of cases)

Transceiver temperature first:

# Cisco NX-OS
show interface ethernet 1/1 transceiver details

# Cumulus Linux
nv show interface swp1 transceiver

⚠️ Sustained above 60°C, the module’s performance degrades. Swapping cables won’t help here — fix the airflow or replace the module.

Then the cable spec. Buying the wrong cable grade is more common than you’d think: a QSFP28 cable used where QSFP56 was needed pins bandwidth at 50G; a certified cable brought it straight back to 100G+. The connectors look identical while the speed differs by 2× — read the label, not the shape.

Physical-layer signal quality:

ethtool <iface> | grep -i link
ethtool -S <iface> | grep -i crc
ethtool -S <iface> | grep -i error
cat /sys/class/infiniband/mlx5_0/ports/1/counters/symbol_error_counter

Verdict: CRC or error counters continuously climbing → the physical layer is the problem. Replace the cable or the optic.

Step 2 — MTU aligned end to end (~30% of cases)

The single highest hit rate on the list — and the same check as step 06 of Part 1:

ping -M do -s 8972 <peer_IP>

• ✅ Replies = MTU is consistent end to end.
• ❌ No reply = some segment is misaligned. Walk it hop by hop: Leaf downlink, uplink, Spine port, VLAN interface. A single path can cross four or five devices, and missing any one of them breaks it.

Configuration commands are in Part 1 step 06 (ip link set <iface> mtu 4200 on hosts, mtu 9216 on switch ports), along with the warning that bigger is not automatically better — race the candidate MTUs with ib_write_bw and keep the winner.

Step 3 — PFC configuration consistency (~20% of cases)

RoCE runs on a lossless network, and lossless is delivered by PFC. The most common PFC pit of all: it’s only enabled on one side. Both switches (Spine and Leaf) need it on, and both rx and tx must be enabled.

Cisco NX-OS:

show interface priority-flow-control

# Port       Mode     Oper(VL bmap)   RxPPP     TxPPP
# ---------  -------  -------------   -------   -------
# Eth1/1     ON       On(8)           4885      3709920
# Eth1/2     ON       On(8)           0         1073455
# Eth1/3     OFF      Off             0         0

Verdict: RxPPP and TxPPP continuously climbing → PFC is firing constantly, which is not a healthy state.

Cumulus Linux:

nv show interface swp1 qos roce status

# pfc
#   pfc-priority      3
#   rx-enabled        yes
#   tx-enabled        yes

Both sides must show rx/tx as on/yes — if either side is off, PFC simply doesn’t work.

⚠️ Occasional PFC triggering is normal. Continuous rapid growth means something is wrong — usually a mismatched ECN threshold or under-provisioned buffer.

Step 4 — PFC storm hunt (~20% of cases)

A PFC storm is sneakier than PFC being off: the configuration looks perfectly correct, but throughput won’t climb.

Locate the storming port — look for the ports with high TxPPP:

show interface priority-flow-control

The usual triggers:

• Incast — AllReduce-style many-to-one traffic overwhelms the buffer.
• RDMA and TCP mixed on one port — TCP retransmissions and PFC pause frames stack up on each other.
• ECN threshold set too high — ECN never gets the chance to slow senders down, so PFC fires first.

Confirm on the host side:

ethtool -S <iface> | grep pause

Once confirmed, the fixes point in three directions: check whether the ECN threshold is too high, keep PFC and TCP off the same physical port, and physically isolate RDMA traffic from everything else.

Step 5 — ethtool counter deep-read (~10% of cases)

The first four steps found nothing — time to read the NIC’s hardware counters.

The counter names below apply to the NVIDIA mlx5 driver; other vendors name them differently.

ethtool -S <iface> | grep -E "drop|discard|error|miss"

The ones to watch:

PFC and RDMA statistics:

ethtool -S <iface> | grep -i pause
ethtool -S <iface> | grep rdma

⚠️ Totals alone tell you nothing — watch the delta while traffic is running. On a healthy link the PFC counters should barely move. If pause counters keep climbing, go back to steps 3 and 4.

Triage summary

The first three steps already cover ~70% of cases.

The order is by frequency, not a mandatory sequence. In practice: if the first three steps come up empty, going straight to a packet capture is often more efficient than grinding through steps 4 and 5 — the later steps need much more information to be conclusive.

Most RDMA packet-loss tickets end at the most basic configuration, not anything deep. The hard part of the triage isn’t that any operation is complex — it’s doing them in the right order. Start with the complicated suspects and you burn hours; start with the 5-minute checks and most tickets never get past step 3.