Blog ▸ Why 8 GPUs Is Not 8x One GPU: Scaling Efficiency in Practice
GPU Infrastructure
At TP=8, communication overhead consumes 25 to 44% of potential speedup. Eight GPUs deliver roughly four to six times one GPU while the invoice scales linearly.
Why 8 GPUs Is Not 8x One GPU: Scaling Efficiency in Practice
GPUaaS.com Team
GPU Infrastructure
September 1, 2026
No items found.
At TP=4 on PCIe systems, communication consumes 40 to 50% of inference time. At TP=8, you lose 25 to 44% of the potential speedup to communication overhead.
That is normal. Not a misconfiguration, not a tuning problem, not something a better NCCL setting fixes. It is what happens when eight GPUs have to agree with each other on every layer.
Key takeaways
Eight GPUs deliver roughly four to six times one GPU's throughput. The invoice scales linearly, the output does not
Mixed precision scales worse than FP32. Faster compute does not reduce communication volume, so communication becomes a larger share of total time
Superlinear scaling is real on memory-bound workloads. Two A100s to four has been measured above 2x throughput
Running a 7B model across four GPUs is usually a mistake. Four independent single-GPU instances serve more total throughput
Ask a provider for measured MFU at the GPU count in the quote, not peak specs multiplied by card count
◆ MEASURED PARALLEL EFFICIENCY AT SCALE
GPU count
FP32 efficiency
Mixed precision efficiency
8 (baseline)
1.00
1.00
16
0.91
0.89
32
0.85
0.82
64
0.71
0.67
Source: HPC AI500 image classification scaling experiments, NCCL double binary tree topology, 8-GPU single node as baseline
◆ THE MECHANISM IS SYNCHRONIZATION
The slowest GPU sets the pace for every step
So eight GPUs deliver somewhere between four and six times one GPU's throughput, depending on interconnect and workload. The invoice scales linearly. The output does not.
Tensor parallelism splits each transformer layer across GPUs, which means every layer boundary becomes a communication event where all participating GPUs must exchange results before proceeding. Add GPUs and you add both the volume of data being exchanged and the number of participants that have to finish before anyone moves on. The slowest GPU sets the pace for every step.
NVLink is what makes this tolerable rather than prohibitive. At 900 GB/s, communication stays small relative to compute. On PCIe at roughly 64 GB/s bidirectional, it does not, which is why NVLink is effectively mandatory for tensor parallelism beyond TP=2 and why PCIe scaling goes visibly sublinear at 4x and above.
◆ FINDING ONE
Mixed precision scales worse than FP32
Parallel efficiency on image classification runs 0.91, 0.85, and 0.71 at 16, 32, and 64 GPUs in FP32. The mixed-precision implementation lands lower at each point: 0.89, 0.82, and 0.67. This looks backwards until you see why. Faster compute does not reduce the communication volume, so communication becomes a larger share of total time. Optimizing the compute half of the workload makes the scaling curve worse, not better. A team that adopts FP8 and then scales out should expect efficiency to drop rather than hold.
◆ FINDING TWO
Superlinear scaling is real, in one specific case
Going from two A100s to four on a memory-bound serving workload has been measured at more than 2x throughput improvement. That happens because the constraint being relieved is memory rather than compute. When a workload is memory-bound, additional GPUs remove a bottleneck rather than adding a coordination cost, and the scaling curve bends the right way. This is the exception, and identifying whether a workload is memory-bound or compute-bound before scaling out is what determines which curve applies.
◆ FINDING THREE
Sometimes the answer is N separate single-GPU jobs
Running a 7B model across four GPUs is a common configuration and usually a mistake. The communication overhead outweighs any benefit, and four independent single-GPU instances serve more total throughput than one four-way tensor-parallel deployment. Multi-GPU is for models that genuinely do not fit, or for throughput a single card cannot sustain. It is not a default.
25-44%
of potential speedup lost to communication overhead at TP=8, which is expected behaviour rather than a configuration fault
◆ PIPELINE PARALLELISM HAS A DIFFERENT FAILURE MODE
Each GPU idle 75% of the time on a single request
With PP=4 on an 80-layer model, each GPU holds twenty consecutive layers and requests flow through like an assembly line. For a single request, only one GPU is working at any moment. Each GPU sits idle 75% of the time waiting its turn. Continuous batching hides most of this in production, since GPU 0 can process the next request while GPU 1 handles the previous one, but the underlying inefficiency is why pipeline parallelism suits high-throughput batch serving far better than low-latency single-request work.
Tensor parallelism demands tight, high-bandwidth intra-node communication and its achievable degree is capped directly by interconnect bandwidth and latency. Data parallelism tolerates latency better but still needs efficient all-reduce at scale. Hybrid approaches, tensor parallel inside a node and pipeline or data parallel across nodes, exist because the two constraint profiles are different and the topology should match.
◆ THE PROCUREMENT CONSEQUENCE
Linear spend for sublinear gains
Cluster sizing done on linear assumptions overpays. A team that needs 6x single-GPU throughput and provisions six GPUs will not get there. A team that provisions eight and budgets for 8x will find roughly 5x arriving and a capacity plan built on a number that never existed. Poor interconnect design specifically means linear GPU spend for sublinear throughput gains, which is a procurement failure disguised as a technical one.
The metric that closes this gap is Model FLOP Utilization, measured across the actual target cluster size before committing to it. Not peak specs, not vendor throughput claims at an unspecified scale, not a single-GPU number multiplied by GPU count. MFU at the size being bought.
That is also the specific question worth putting to a provider before signing. What is the measured scaling efficiency on this fabric at the GPU count in the quote. A provider running NVLink-connected SXM hardware can usually answer it. One selling PCIe-connected capacity as though it scales the same way often cannot, and the difference shows up in the first multi-node run rather than in the contract.
This is why L40S and RTX PRO 6000 are single-GPU parts regardless of how many you put in a chassis, and why H100 SXM and B200 SXM remain the answer for anything genuinely distributed.
Get quoted on a fabric that matches your parallelism strategy.
Not a card count multiplied by a single-GPU number. No buyer fees. For single GPUs, packet.ai handles self-serve access with 24/7 human support.
Roughly four to six times a single GPU, depending on interconnect and workload. At TP=8, communication overhead consumes 25 to 44% of the potential speedup. That is expected behaviour rather than a tuning fault.
No, it usually makes scaling efficiency slightly worse. Measured parallel efficiency at 64 GPUs runs 0.71 in FP32 against 0.67 for mixed precision. Faster compute does not reduce communication volume, so communication becomes a larger proportion of total time.
Yes, on memory-bound workloads. Moving from two A100s to four has been measured above 2x throughput, because additional GPUs relieve a memory constraint rather than adding coordination cost. Determining whether a workload is memory-bound or compute-bound before scaling decides which curve applies.
Usually not. Running a 7B model across four GPUs means communication overhead outweighs the benefit, and four independent single-GPU instances serve more total throughput. Multi-GPU is for models that do not fit or throughput a single card cannot sustain.
Measured Model FLOP Utilization on their fabric at the GPU count in the quote. Not peak specs, not a single-GPU figure multiplied by card count. Providers running NVLink-connected SXM hardware can usually answer. The difference surfaces in the first multi-node run otherwise.
Last reviewed: 2 September 2026. Tensor parallel communication overhead figures from PreMAI's multi-GPU LLM inference parallelism guide, July 2026. Parallel efficiency scaling data from HPC AI500 image classification experiments using NCCL double binary tree topology. Superlinear memory-bound scaling from S-LoRA multi-GPU tensor parallelism benchmarks. Interconnect and configuration guidance from Spheron's vLLM production deployment guide and FPT AI's GPU selection analysis. Browse current GPU cluster availability on GPUaaS.com.
