NCCL and NVLink Notes

Wed, 05 Aug 2026 21:00:00 +0800

Note: this post is a re-write based on articles found on the internet, cross-checked against NVIDIA’s NCCL documentation and published NVLink specifications, and kept here for future reference. This revision restructures the post around a much more complete source article covering NCCL’s core concepts, algorithms, protocols, tuning, and troubleshooting.

1. What NCCL is, and why a memcpy needs a library

Training a model across many GPUs is, mechanically, a loop of compute and synchronization. With 8 GPUs data-parallel training a 7B model, each card holds ~14 GB of bf16 gradients after every backward pass, and all 8 copies must be merged into one identical result on every card — hundreds of thousands of times over a run. The standard tool for this is NVIDIA’s NCCL (NVIDIA Collective Communications Library, usually pronounced “Nickel”). NCCL is not a training framework and not a scheduler; it is the layer that moves tensors between GPUs, with topology awareness and per-path optimization for PCIe, NVLink, NVSwitch, InfiniBand, RoCE, and plain Ethernet. Every “data parallel”, “tensor parallel”, or “parameter sharding” feature in PyTorch, TensorFlow, JAX, Megatron-LM, DeepSpeed, Horovod, or vLLM eventually lands on the small set of primitives this post walks through — in training and in large-model inference (tensor/pipeline/expert parallel, KV-cache traffic).

Nccl-Tests on Council of Elrond

NCCL and NVLink Notes

1. What NCCL is, and why a memcpy needs a library