CollNetDirect all-reduce · 4 GPUs
A = ⊕aᵢ
B = ⊕bᵢ
C = ⊕cᵢ
D = ⊕dᵢ
2 nodes · 2 heads / rails
Previous
Next
Input
Step 0 of 4 · Local vectors