TP, PP, DP: How Large Models Actually Use Multiple GPUs

Tue, 25 Aug 2026 12:00:00 +0800

Note: this post started from several explainers found via web search, was cross-checked against the Megatron-LM paper and the vLLM documentation, and adds the memory math, communication costs, and decision rules those explainers leave out. It is the deployment-side companion to the NCCL and NVLink notes, which cover what actually moves on the wire underneath these strategies.

When you deploy a large model, you keep seeing configurations like “Qwen-72B, 8 GPUs” or “Qwen-32B, 2× L20” — and the natural question is: how is one model split across several cards, and how do the cards cooperate? Multi-GPU is not “the VRAM adds up.” The GPUs have to be organized, and there are three basic ways to organize them — Tensor Parallelism (TP), Pipeline Parallelism (PP), and Data Parallelism (DP) — each solving a different problem.

LLM Inference on Council of Elrond

TP, PP, DP: How Large Models Actually Use Multiple GPUs