Perftest_RDMA_Benchmarking

Sat, 18 Jul 2026 20:00:00 +0800

What is perftest?

perftest is the standard micro-benchmark suite for RDMA — it validates and benchmarks InfiniBand, hardware RoCE, and Soft-RoCE links, not ordinary TCP networking. Each tool measures one RDMA verb:

Test	Measures	Real-world workloads it models
`ib_write_bw`	RDMA Write bandwidth	Data replication, storage writes, GPU/bulk data transfers
`ib_write_lat`	RDMA Write latency	Updating remote memory — small one-sided writes
`ib_read_bw`	RDMA Read bandwidth	Distributed storage and database reads, remote memory access
`ib_read_lat`	RDMA Read latency	Fetching remote memory — storage/database-style lookups
`ib_send_bw`	Send/Receive bandwidth	MPI messages, RPC, request queues
`ib_send_lat`	Send/Receive latency	Small request/response round trips — RPC and MPI latency
`ib_atomic_bw`	Atomic operation rate	Distributed locks and counters
`ib_atomic_lat`	Atomic latency	Synchronization primitives

Write, Read, and Send are different operations

The grouping in the table is not accidental — the three verbs have different semantics:

RoCE_Cumulus_Linux

Fri, 17 Jul 2026 20:00:00 +0800

Part 1: RoCE and RDMA fundamentals

In the context of NVIDIA/Cumulus Linux and HPC networking, RoCE stands for:

RoCE = RDMA over Converged Ethernet

It is widely used in AI clusters, HPC, GPU clusters, storage networks, and low-latency data centers.

What is RoCE?

RoCE allows RDMA (Remote Direct Memory Access) to run over an Ethernet network.

Normally, when one server sends data to another:

Application
 |
 v
TCP/IP Stack
 |
 v
Kernel
 |
 v
NIC
 |
 v
Ethernet