NCCL and NVLink: Collective Communication and the Two Domains of an AI Cluster

Wed, 05 Aug 2026 21:00:00 +0800

Note: this post is a re-write based on articles found on the internet, cross-checked against NVIDIA’s NCCL documentation and published NVLink specifications, and kept here for future reference.

1. NCCL: the communication primitives under every training job

Training a model across many GPUs is, mechanically, a loop of compute and synchronization. With 8 GPUs data-parallel training a 7B model, each card holds ~14 GB of bf16 gradients after every backward pass, and all 8 copies must be merged into one identical result on every card — hundreds of thousands of times over a run. The standard tool for this is NVIDIA’s NCCL (NVIDIA Collective Communications Library). NCCL is not a training framework and not a scheduler; it is the layer that moves tensors between GPUs. Every “data parallel”, “tensor parallel”, or “parameter sharding” feature in the frameworks above it eventually lands on the small set of primitives this section walks through.

GPU on Council of Elrond

NCCL and NVLink: Collective Communication and the Two Domains of an AI Cluster

1. NCCL: the communication primitives under every training job