Ring all-reduce · 4 GPUs
A = ⊕aᵢ
B = ⊕bᵢ
C = ⊕cᵢ
D = ⊕dᵢ
i = 0…3 · ⊕ = NCCL reduction
Previous
Next
Input
Step 0 of 6 · Local data