Network on Council of Elrond

RoCE QoS Concepts and Packet Examples

Fri, 24 Jul 2026 13:00:00 +0800

There are many docs that use different terms on QoS — PCP vs CoS, for example, are in fact the same thing. This doc consolidates the QoS concepts and terminology behind RoCEv2 deployments, with decoded packet examples.

It is the concepts companion to two other posts on this blog: End-to-End RoCEv2 Configuration: Cisco Nexus, Cumulus Linux, and ConnectX is the configuration guide that turns these concepts into working templates, and RDMA Performance Tuning covers tuning and packet-loss troubleshooting once the fabric is up.

RDMA Performance Tuning: From Fresh Install to Line-Rate Bandwidth

Thu, 23 Jul 2026 21:00:00 +0800

The number-one complaint with RDMA is simple: the bandwidth won’t fill up. A 100G card pushing only 60G, two days spent checking drivers, optics, and the peer’s configuration — and in the end the fix is a single PCIe parameter.

When RDMA underperforms, 90% of the time the cause is not exotic. It’s some factory default that was never touched: the switch buffer carved too small, the PCIe read request size left at its tiny default, the MTU misaligned on one segment of the path. None of these is hard to fix on its own — the hard part is knowing which one to check.

EVE-NG_6_SONiC_Virtual_Switch_Installation

Thu, 23 Jul 2026 20:00:00 +0800

SONiC (Software for Open Networking in the Cloud) publishes a virtual-switch build (SONiC-VS) that boots as a QEMU VM, which makes it a perfect lab guest for EVE-NG. This post walks through installing SONiC-VS on EVE-NG 6 using the built-in Sonic Switch template, validating the first boot with real console captures, and finishes with a command reference for the SONiC CLI — the commands you actually reach for once the switch is up: interfaces, routing, logs, config, and user management.

Cheat_Sheet

Tue, 21 Jul 2026 04:00:00 +0800

Every verification and troubleshooting command from the lab posts on this blog — plus a few general-purpose Linux staples — collected in one place and organized by vendor, because that’s how you reach for a cheat sheet: you’re on a box, and you need to know what to run. Each subsection links back to the post it came from where one exists, and explains what each command shows and what a healthy result looks like — a command you can’t interpret is just noise.

Arista_VXLAN_EVPN_Lab

Tue, 21 Jul 2026 00:30:00 +0800

A VXLAN fabric has to answer one question before any host can talk: how does BUM traffic (Broadcast, Unknown unicast, Multicast) reach every remote VTEP? There are only two data-plane answers — head-end replication (HER, also called ingress replication) or an IP multicast underlay — and two control-plane ways to drive each of them (static configuration or BGP EVPN).

This lab builds the same Arista fabric four times, once per combination:

End-to-End RoCEv2 Configuration: Cisco Nexus, Cumulus Linux, and ConnectX

Mon, 20 Jul 2026 04:30:00 +0800

A RoCEv2 fabric is not complete when the switch merely forwards UDP port 4791. The endpoints and every switch hop must agree on a traffic-class contract: DSCP classification, internal priority, egress queue, PFC priority, ECN thresholds, CNP handling, MTU, and DCQCN behavior.

This guide turns that contract into two switch implementations and one host implementation:

Cisco Nexus 9000 with NX-OS Modular QoS CLI (MQC)
NVIDIA Spectrum with Cumulus Linux and NVUE
Linux servers with NVIDIA ConnectX-5/6/7 adapters

The examples use DSCP 24 for RoCE data, DSCP 48 for CNP, and PFC priority 3. Those values are a design choice, not a protocol requirement. If your environment uses DSCP 26, a different PFC priority, or a vendor profile, change the entire path consistently.

Arista_EOS_RoCEv2_Lossless_Config

Mon, 20 Jul 2026 04:00:00 +0800

On Cisco NX-OS, enabling PFC for RoCE normally involves several MQC (Modular QoS CLI class-map -> policy-map -> system qos) layers. On Arista EOS, basic PFC can be enabled directly on an interface with two commands:

priority-flow-control on
priority-flow-control priority 3 no-drop

Note: these commands cannot be executed in an EVE-NG lab. PFC is a hardware feature — the pause frames are generated and honored by the switching ASIC — and the virtual vEOS-lab image has no such hardware, so EOS rejects the configuration outright. Real output from a vEOS-lab node:

SFP, QSFP, Fiber Types

Sun, 19 Jul 2026 18:00:00 +0800

Choosing an optical transceiver is harder than matching the number printed on the switch port. A working link has to align the form factor, host lane rate, Ethernet standard, optical organization, fiber type, connector, wavelength, FEC mode, loss budget, and platform support at both ends.

The first clarification is terminology: SFP is a specific one-lane form factor, not the general name for every removable optic. Engineers often say “SFP” informally, but 400G and 800G modules are normally QSFP112, QSFP-DD, QSFP-DD800, or OSFP.

VXLAN EVPN Architecture

Sun, 19 Jul 2026 12:00:00 +0800

VXLAN EVPN combines a scalable Layer 2 data plane with a standards-based MP-BGP control plane. VXLAN carries Ethernet frames across a routed IP fabric; EVPN distributes endpoint, subnet, and tunnel-reachability information so the fabric does not have to discover everything by flooding.

Contents — 22 sections

Standards and implementation scope

VXLAN EVPN is not defined by one document. The architecture is assembled from a data-plane encapsulation, an EVPN control plane, a mapping between EVPN and network-virtualization overlays, and later IRB and prefix-route extensions:

VXLAN_EVPN_Cumulus_Lab_Test

Sun, 19 Jul 2026 02:00:00 +0800

This lab builds a VXLAN EVPN fabric on Cumulus Linux 5.4: one BGP spine, an MLAG leaf pair (leaf1/leaf2) acting as a single logical VTEP, and a standalone leaf (leaf3). Linux1 is dual-homed to the MLAG pair over an LACP bond, while Linux21 and Linux22 share a stretched VLAN with a distributed anycast gateway.

Lab requirements

VLAN 100: Linux1 192.168.100.10/24, gateway 192.168.100.1
VLAN 121: Linux21 192.168.121.21/24, Linux22 192.168.121.22/24, anycast gateway 192.168.121.1 on all leaves

Lab environment

This lab runs in EVE-NG. The four switches (spine, leaf1, leaf2, leaf3) are Cumulus VX 5.4 nodes, and the three hosts (Linux1, Linux21, Linux22) are lightweight Alpine-style Linux nodes configured through /etc/network/interfaces and OpenRC’s rc-service. Any emulator that boots Cumulus VX 5.4 with the port mapping in section 2 will work; adjust the host commands if your Linux image uses a different init system or network configuration method.

Perftest_RDMA_Benchmarking

Sat, 18 Jul 2026 20:00:00 +0800

What is perftest?

perftest is the standard micro-benchmark suite for RDMA — it validates and benchmarks InfiniBand, hardware RoCE, and Soft-RoCE links, not ordinary TCP networking. Each tool measures one RDMA verb:

Test	Measures	Real-world workloads it models
`ib_write_bw`	RDMA Write bandwidth	Data replication, storage writes, GPU/bulk data transfers
`ib_write_lat`	RDMA Write latency	Updating remote memory — small one-sided writes
`ib_read_bw`	RDMA Read bandwidth	Distributed storage and database reads, remote memory access
`ib_read_lat`	RDMA Read latency	Fetching remote memory — storage/database-style lookups
`ib_send_bw`	Send/Receive bandwidth	MPI messages, RPC, request queues
`ib_send_lat`	Send/Receive latency	Small request/response round trips — RPC and MPI latency
`ib_atomic_bw`	Atomic operation rate	Distributed locks and counters
`ib_atomic_lat`	Atomic latency	Synchronization primitives

Write, Read, and Send are different operations

The grouping in the table is not accidental — the three verbs have different semantics:

RoCE_Cumulus_Linux

Fri, 17 Jul 2026 20:00:00 +0800

Part 1: RoCE and RDMA fundamentals

In the context of NVIDIA/Cumulus Linux and HPC networking, RoCE stands for:

RoCE = RDMA over Converged Ethernet

It is widely used in AI clusters, HPC, GPU clusters, storage networks, and low-latency data centers.

What is RoCE?

RoCE allows RDMA (Remote Direct Memory Access) to run over an Ethernet network.

Normally, when one server sends data to another:

Application
 |
 v
TCP/IP Stack
 |
 v
Kernel
 |
 v
NIC
 |
 v
Ethernet

The CPU has to copy memory, handle interrupts, process TCP/IP, and switch between user and kernel space. This consumes CPU cycles and adds latency.

TCP_Three_Way_Handshake_and_Four_Way_Termination

Thu, 16 Jul 2026 00:00:00 +0800

Study notes rewritten in English.

TCP basics

What is TCP?

TCP is a connection-oriented, reliable, byte-stream transport-layer protocol.

Connection-oriented – strictly one-to-one. Unlike UDP, one host cannot broadcast to many hosts over a single TCP connection.
Reliable – no matter what happens on the link, TCP guarantees a segment eventually reaches the receiver.
Byte stream – the OS may split a user message into several TCP segments. Segments are ordered: if the previous segment hasn’t arrived, a later one that did arrive cannot be handed up yet; duplicate segments are dropped.

IP is unreliable — it doesn’t guarantee delivery, ordering, or integrity. If you need reliability, the transport-layer TCP has to provide it.

How_Linux_Sends_and_Receives_Network_Packets

Sat, 11 Jul 2026 07:00:00 +0800

Study notes rewritten in my own words

Network models

To let all kinds of devices talk to each other over a network, the ISO defined the OSI reference model with 7 layers. Each layer has a single job:

Application – provides a uniform interface to programs
Presentation – converts data into a format another system understands
Session – establishes, manages and terminates sessions
Transport – end-to-end data transfer
Network – routing, forwarding, fragmentation
Data link – framing, error detection, MAC addressing
Physical – transmits frames over the physical medium

OSI is mostly a conceptual model — it never shipped a concrete implementation. What we actually use is the simpler TCP/IP model with 4 layers, and the Linux stack is built to match it:

PIM Sparse Mode in Detail

Mon, 20 Nov 2023 12:00:00 +0800

A step-by-step walk through the PIM Sparse Mode (PIM-SM) control plane — from a receiver’s IGMP report through shared-tree construction, source registration, the native source tree, and the optional shortest-path-tree switchover — then how to make the Rendezvous Point redundant with Anycast RP. For the Designated Router versus Assert election and multicast-ECMP behavior, see the companion post PIM DR, Assert, and Multicast ECMP.

1. Overview

Protocol Independent Multicast Sparse Mode (PIM-SM) builds multicast distribution trees only where receivers explicitly request traffic. It is intended for networks where multicast receivers are sparsely distributed.

PIM DR, Assert, and Multicast ECMP

Sat, 11 Nov 2023 12:00:00 +0800

Several questions come up whenever two PIM routers share a receiver VLAN over equal-cost paths: if the DR and the Assert winner are different routers, which one forwards the multicast? Does a single stream ever spread its packets across both paths? If you change nothing, do 500 streams split evenly across the two routers—or does that take configuration? And what changes if the switch becomes a Layer 3 router? This post works through each, with the underlying PIM mechanics and a clear line between what the standards mandate and what is vendor-specific.

Load_Balancing

Sun, 20 Aug 2023 21:29:04 +0800

LoadBalancing

Load balancing (Server Load Balance) means to balance and distribute the load (work tasks) to multiple operating units for execution, so that the entire system can complete the task together.

Load balancing provides a cheap and effective method to expand the bandwidth of network devices and servers, increase throughput, enhance network data processing capabilities, and improve network flexibility and availability. According to the different implementation purposes of load balancing, load balancing can be divided into two types. One is task sharing, in which a large amount of concurrent access or data traffic is shared among multiple devices for separate processing, and each device completes a relatively complete request response process; the other is collaborative computing, which shares a heavy-load computing task. Perform parallel processing on multiple devices. After each device completes processing, the results are summarized and calculated.

CDN_Cache_Cluster_communication

Mon, 16 May 2022 12:29:04 +0800

Cache Cluster Communication

The main goal of collaborative interaction within the Cache server cluster is to establish a good communication channel between each server node, communicate in a timely manner about the content cache on the server, and provide users with a good service experience through collaboration between servers in the cluster.

Communication between Cache server clusters can be divided into two categories: loose coupling and tight coupling. Among them, loosely coupled Cache communication protocols based on network messages include: ICP, HTCP, Cache Digest, Cache Pre-filling, etc. Tightly coupled Cache communication protocols managed by specific data structures are represented by CARP. Below is just a brief introduction to these communication protocols, focusing on their working principles, advantages and disadvantages.

Low_Latency_Network_Architecture

Thu, 10 Mar 2022 20:04:01 +0800

Originally published in March 2022. Substantially revised in July 2026: the architecture analysis was rebuilt around current exchange designs (Eurex Co-location 2.0, CME MSGW, Nasdaq NY11-4), the sequencer pattern, traffic-plane segregation, and the hybrid FPGA fast path, with refreshed vendor and latency data. Sources are linked inline; latency figures from vendor datasheets are best-case specs, not independent measurements.

1. What a trading system architecture optimizes for

Below is a simple sketch of the key components of an exchange trading system. Legacy systems ran everything on centralized servers; the network carries market data, order routing and clearing.

CDN_Notes

Sat, 08 Jan 2022 02:36:13 +0800

Content Delivery Network

Below is all my personal understandings

Full Name :Content Delivery Network or Content Ddistribute Network

Basically, CDN solution solves a common issue: There will always be a bottleneck on the internet, due to different service providers, or geo locations, etc. Which impact the network performance. On the other hand, Latency control is one of the most important indicators nowadays. With CDN, You can deliver a more reliable and stable network by putting certain Servers in certain places in the internet, and building a intelligent network on top of the internet. CND can redirect customer reuquests to the fastest server(maybe not the closest), based on network throughput, response time, latency, loadbalancing status, physical distance(main factor that impact latency).