B300 Cost Per Million Tokens: Measured, Not Estimated

◆ MEASURED COST PER MILLION TOKENS, B200 VS B300

Source: SemiAnalysis InferenceX independent benchmarks, 8k/1k sequence at FP4, 2026

◆ THE RANKING FLIPS

Same two chips, same model, opposite conclusion

Same two chips, same model, opposite conclusion. The only thing that changed is how fast each user is being served.

On GLM 5/5.1 the picture is different again. At 37 tokens per second per user, B200 runs $0.29 per million against B300's $0.49. At 63, it is $0.43 against $0.67. At 88, $0.62 against $0.92. B200 is cheaper at every point measured across that band, by 49 to 68%.

There is no single cost per million tokens for B300. There is a cost for a specific model at a specific interactivity target on a specific software version, and the ranking flips depending on all three.

◆ WHY IT FLIPS

B300 only wins when the workload is compute-bound

The mechanism behind the flip is straightforward once the numbers are in front of you. B300's FP4 advantage comes from three places: all 160 streaming multiprocessors enabled against 148 harvested on B200, higher clocks under the larger power envelope, and tensor core work specific to NVFP4. Every one of those only helps when the workload is compute-bound.

At low interactivity and small batch, the workload is not compute-bound. It is bandwidth-bound, and both chips run 8 TB/s. B300 is burning a higher hourly rate to deliver throughput the workload cannot use. That is the entire story behind the $0.06 versus $0.08 result.

At high interactivity and large batch, the workload becomes compute-bound, and B300's FP4 advantage lands. On Llama 3.3 70B at FP4, B300 delivers roughly 16,500 tokens per second against B200's 10,000. Cost per million falls to $0.055 from $0.074.

◆ MEMORY HEADROOM SHOWS UP AS TAIL LATENCY

Not as a throughput number

Memory headroom is the second lever, and it shows up as tail latency rather than as a throughput number. A 70B model in FP16 needs about 140GB for weights alone. At batch size 32 on B200, what remains for KV cache is tight enough that paged attention starts evicting blocks to CPU memory once context lengths pass roughly 32K. Every eviction that has to be swapped back adds latency to that request. B300's larger capacity leaves around 148GB of KV cache headroom under the same conditions, which pushes that eviction point much further out.

This is where the cost comparison stops being a single number. A workload that never hits eviction pressure sees B300 as a more expensive chip doing the same job. A workload that hits it constantly sees B200 as a chip that quietly fails its latency target.

◆ THE BREAK-EVEN THAT ACTUALLY MATTERS

Fleet-level substitution, not single-GPU arithmetic

The break-even math on a single GPU is unforgiving. Covering B300's premium purely through lower cost per token on Llama 3.3 70B requires serving roughly 770 million tokens per day on one GPU, about 9,000 tokens per second sustained. That is close to 90% of what a single B200 can do on that workload. Nobody runs a single GPU at 90% before adding capacity.

The break-even that actually matters is different, and it is the one worth planning around. The moment a team would otherwise add a second B200 to handle demand, one B300 often wins instead. Fleet-level substitution, not single-GPU arithmetic.

◆ READ VENDOR BENCHMARKS CAREFULLY

Every configuration choice is a thumb on the scale

Vendor benchmarks deserve a specific kind of caution here. NVIDIA's GTC cost-per-million-token comparison used FP4, multi-token prediction set to 3, and one particular day's data on a DeepSeek 1k/1k sequence. Every one of those choices is legitimate on its own and every one of them favors the result. Change the precision to FP8, which remains a common production choice, or turn MTP off, and the competitive picture moves.

This is not a criticism unique to NVIDIA. It is what benchmark selection does, and it is why an independent framework testing a broad universe of configurations is more useful than any single headline figure.

◆ EVERY NUMBER HERE CARRIES A TIMESTAMP

Software moves faster than hardware

Software movement complicates the whole comparison further, in a direction that helps buyers. GB300 NVL72 throughput rose up to 2.7 times in a single MLPerf round through software optimization alone, with no hardware change. B200 inference throughput improved roughly five times since launch through TensorRT-LLM and Dynamo work, pushing effective cost as low as $0.02 per million tokens on some workloads.

Any cost-per-token number carries a timestamp whether it prints one or not. For the underlying method behind these calculations, see B200 cost per million tokens, measured. For B300 specs and configuration options, see the B300 buyer's guide.

The practical read: unless a model exceeds B200's memory capacity, or the workload sits at high interactivity where B300's FP4 throughput actually lands, the lower-cost chip usually wins on unit economics. That is a narrower case for B300 than the marketing suggests, and it is a real case rather than a hypothetical one.

◆ FAQ

Frequently asked questions