InfiniBand vs Ethernet in 2026: The Bottleneck Nobody Priced

◆ THE MEASUREMENT THAT SETTLES IT

Not the fabric. The number nobody collects.

The performance gap that premium buys is 5 to 15% on typical training workloads. Whether that trade is worth making depends on one number most teams never measure: what fraction of step time their workload actually spends on communication.

That is the bottleneck nobody prices. Not the fabric. The measurement that tells you which fabric you need.

Worth noting those TCO figures come from Vitex, a vendor selling Ethernet networking gear, so read the exact numbers as directionally useful rather than neutral. The direction itself is corroborated widely: InfiniBand carries a premium in the range of $1.2 million on hardware and $1.4 million on three-year TCO at 512 GPUs.

◆ WHERE THE GAP IS, AND WHERE IT ISN'T

Sources: Spheron GPU networking decision guide (April 2026), WWT independent test via Servnet UK analysis (August 2026), Vitex InfiniBand vs Ethernet TCO model

◆ A GAP ON ALL-REDUCE IS NOT A GAP ON TRAINING TIME

It only becomes one if all-reduce dominates the step

The raw performance difference is real and measurable. NCCL all-reduce on eight H100 SXM5 GPUs achieves roughly 350 GB/s of effective bus bandwidth on InfiniBand NDR 400G. The same cluster on 400GbE RoCEv2 gets 270 to 290 GB/s. That is a 20 to 25% gap on the collective operation itself.

But a 20 to 25% gap on all-reduce is not a 20 to 25% gap on training time. It only becomes that if all-reduce dominates the step. Most workloads it does not.

Across full training runs, properly tuned RoCEv2 delivers 85 to 95% of InfiniBand throughput. On 64-plus GPU clusters training 100B-plus parameter models, InfiniBand typically delivers 15 to 20% better throughput. On 8 to 32 GPU setups, or models under 70B, well-tuned RoCEv2 comes out comparable.

◆ FOR INFERENCE THE PICTURE CHANGES COMPLETELY

The finding most likely to save someone money

An independent World Wide Technology test measured end-to-end performance delta on real generative AI and inference workloads at under 1%, specifically 1.02% in most cases. Spec-sheet microsecond gaps mostly do not translate into job completion differences on inference and generative serving.

If the cluster is serving rather than training, the InfiniBand premium is very hard to justify on performance grounds.

◆ WHY THE TRAINING GAP EXISTS

Flow control and routing, both closing

InfiniBand uses credit-based flow control, which is lossless by design. RoCEv2 achieves lossless behaviour through priority flow control and explicit congestion notification working together, tuned with DCQCN. Both are required. PFC alone produces head-of-line blocking. ECN alone produces packet drops at congestion. Getting that configuration right is real work, and getting it wrong is expensive: on a 1,024-GPU H100 cluster, every 1% packet drop costs roughly $250,000 in lost training time per week.

InfiniBand also has adaptive routing built into the fabric. Ethernet relies on ECMP and flowlet switching, which can create hotspots under static hashing. Adaptive load balancing on Tomahawk 4 and 5 silicon rebinds flows based on real-time congestion, and a well-run fabric can target above 90% utilisation. That is a design target rather than a measured guarantee, and worth treating as such.

◆ THE PROPERTY THAT MATTERS MOST GETS LEAST ATTENTION

Tail latency, not average latency

Tail latency under contention is what determines AllReduce completion time across thousands of GPUs, because the entire cluster waits for the last node to finish each synchronisation. Average latency is not the metric. The worst case is. This is the same mechanism that makes a single fail-slow node gate an entire 512-GPU training job, covered in multi-node training at 512 GPUs and above, and it is why InfiniBand's graceful behaviour during incast events shows up in job completion time rather than in benchmark averages.

◆ THE SCALE ARGUMENT HAS LARGELY COLLAPSED

24,000 GPUs on Ethernet

Meta has described training its largest models over a RoCE Ethernet fabric on a 24,000-GPU cluster. That is a size the industry previously assumed required InfiniBand. Meta, Microsoft, and AWS have all converged on RoCEv2, primarily for operational reasons rather than cost.

Those operational reasons are worth weighing properly. InfiniBand requires specialised operators, separate infrastructure, and a single-vendor supply chain. RoCEv2 runs on Ethernet and BGP that most infrastructure teams already operate. For a team without dedicated fabric engineers, that difference is larger than it looks on a spreadsheet.

Spectrum-X sits between the two and is worth knowing about. It combines Spectrum-4 switches with ConnectX-7 NICs and proprietary adaptive routing, closing roughly 80 to 90% of the gap with InfiniBand NDR on NCCL all-reduce while running standard 800GbE hardware. The adaptive routing benefits require both ends to be Spectrum-X, but the switches themselves interoperate with non-NVIDIA Ethernet gear.

◆ THE DECISION RULE

Measure the communication share first

Run NCCL benchmarks on the actual workload and measure what fraction of step time is communication. If communication is a small share, the fabric choice barely matters and Ethernet wins on cost by a wide margin. If communication dominates and the cluster is at frontier training scale with thousands of GPUs, InfiniBand's latency floor and incast behaviour justify the premium, because a small regression multiplied across a very large synchronised job costs more in wasted GPU-hours than the fabric premium.

If the workload is inference, the WWT result suggests the question is close to settled.

That benchmark run takes hours. The decision it informs runs into millions.

◆ FAQ

Frequently asked questions