B300 Cluster Sizing: How Many GPUs Your Workload Actually Needs

◆ LLAMA 405B AT FP8, TWO WAYS

Source: Spheron B300 vs B200 inference cost-per-token analysis, 2026

◆ THE CENTRAL TRAP

Bandwidth is a per-GPU property

Aggregate memory bandwidth is why. Three B200s give 24 TB/s. Two B300s give 16 TB/s. At FP8 and typical batch sizes, 405B inference is memory-bandwidth-bound, so throughput scales with total bandwidth rather than with total capacity. The 50% bandwidth advantage of the larger cluster wins on tokens per second by more than the extra GPU costs.

That is the central trap in B300 sizing. Bandwidth is a per-GPU property. Using B300's memory advantage to reduce GPU count also reduces aggregate bandwidth, and for bandwidth-bound work that is a downgrade you paid a premium to receive.

So the sizing question is not what fits. It is which constraint binds.

◆ START WITH THE MEMORY FLOOR

It sets the lower bound regardless

A 70B model in BF16 needs roughly 140GB for weights alone. A 405B model needs north of 800GB at full precision. Add KV cache and activations on top. The vLLM startup log is the cheapest way to see the real split: on a single GPU serving an 8B model it reports weights, non-Torch memory, PyTorch activation peak, and whatever remains for KV cache as separate lines. That breakdown is what actually determines concurrency.

An HGX B300 node carries eight GPUs at 2.3TB total, connected by a 14.4 TB/s NVLink switch with 800 Gb/s networking. GB300 NVL72 puts 72 GPUs and 36 Grace CPUs in one liquid-cooled rack. Those are the two shapes worth sizing against, because they determine whether tensor parallelism stays inside a node.

◆ WHERE CONCURRENCY SATURATES, MEASURED ON HGX B200

Nemotron Nano 30B at FP8, TP=2 on two GPUs, input 2048 output 512. Source: Vultr inference cookbook concurrency tuning, verified on NVIDIA HGX B200

◆ THROUGHPUT KEEPS RISING PAST USABLE LATENCY

Two correct answers for two different requirements

Then find where concurrency saturates, because that is where adding GPUs stops helping and the benchmark data is unambiguous about it. Nemotron Nano 30B at FP8 with TP=2 saturates around 512 concurrent requests, peaking at 18,829 tokens per second across two GPUs. Nemotron Super 49B at FP8 on a single GPU saturates around 64 concurrent, peaking at 3,816 tokens per second.

Note the shape of that. Throughput keeps rising well past the point where latency becomes unacceptable. A cluster sized for peak tokens per second will miss a latency target badly, and a cluster sized for latency will look underutilised on a throughput dashboard. Both are correct sizing, for different requirements.

The single-GPU benchmark also shows something worth knowing: throughput at TP=1 oscillates at high concurrency, dropping to 1,586 tokens per second at 256 concurrent before recovering at 512. That is batch scheduling behaviour, not a hardware limit, and it means single-GPU capacity planning needs measurement at the actual target concurrency rather than interpolation between two points.

◆ THEN APPLY THE SCALING PENALTY

A linear count will be wrong

At TP=8, communication overhead consumes 25 to 44% of the potential speedup. That is expected behaviour rather than a fault. Eight GPUs deliver roughly four to six times one GPU's throughput. On B300, NVLink at 14.4 TB/s inside an HGX node keeps this at the better end of that range, and crossing a node boundary moves it to the worse end. The mechanism is covered in why 8 GPUs is not 8x one GPU.

Total GPU count is tensor parallel size multiplied by pipeline parallel size. Four-way tensor parallel across two pipeline stages consumes eight GPUs. Getting that arithmetic wrong is a common way to order half or double the capacity actually needed.

◆ THREE LEVERS THAT MOVE THE ANSWER MORE THAN GPU COUNT

Configuration before capacity

Memory utilisation fraction sets how much of each card goes to weights and KV cache. 0.90 is the default, 0.95 pushes to maximum capacity at some out-of-memory risk, and 0.45 is for running multiple instances on one card. Pushing it up grows the KV cache, which raises how many sequences can batch concurrently, and the gain continues only until the workload becomes compute-bound.

Maximum concurrent sequences should track model size. Memory-constrained large models like GLM-5 and DeepSeek V3.2 want 128 to 256. The balanced default is 512 to 1024. Smaller models chasing maximum throughput can take 2048 to 4096.

KV cache offloading to CPU memory is worth knowing about and usually not worth using. With 192 to 288GB of HBM per card, it is unnecessary except for extreme context lengths, and it adds latency when it engages.

There is a comparison point worth registering on how much headroom modern capacity actually provides. A 685B model at FP8 on an eight-GPU node with 2TB of aggregate memory uses only 83GB for weights, leaving 96% of memory for KV cache. At that scale, capacity stops being the binding constraint entirely and bandwidth and concurrency become the only questions that matter.

The practical protocol: establish the memory floor from weights plus KV cache at target context length, decide whether the workload is bandwidth-bound or compute-bound, size for the latency target rather than peak throughput, apply a 25 to 44% penalty for TP=8, and confirm total count as TP multiplied by PP. Then measure MFU at that count before committing, because the difference between the plan and the measurement is the part that costs money.

◆ FAQ

Frequently asked questions