https://songkou.github.io/tags/distributed-training/Recent content in Distributed Training on Council of ElrondHugoenWed, 09 Sep 2026 12:00:00 +0800https://songkou.github.io/posts/nccl-workflow/Wed, 09 Sep 2026 12:00:00 +0800https://songkou.github.io/posts/nccl-workflow/<blockquote> <p><strong>Note:</strong> this post corrects a wrong mental model I had after studying the two companion posts below. Every claim was cross-checked against NVIDIA&rsquo;s NCCL user guide, the NCCL tuning blog, and the Megatron Core documentation — all linked in the references.</p> </blockquote> <p>After studying <a href="https://songkou.github.io/posts/tp-pp-dp-llm-parallelism/">TP, PP, and DP</a> and <a href="https://songkou.github.io/posts/nvlink-nccl-scaleup-scaleout/">NCCL and NVLink</a>, I initially compressed the process into two steps:</p> <ol> <li>Choose TP, PP, or DP based on workload size and concurrent users.</li> <li>Detect the network topology and choose something like Reduce or AllReduce to synchronize GPUs.</li> </ol> <p>That description mixes decisions made at different layers. <strong>The application or framework decides how to partition the work and what communication result it needs. NCCL decides how to carry out the requested communication.</strong></p>