LLM Inference Cost on H100 in 2026: What It Actually Costs to Serve a Model at Scale

◆ THE BANDWIDTH CEILING

Memory, not compute, sets the limit

H100's throughput ceiling comes from memory bandwidth, not compute. 3.35 TB/s serves a 70B model at roughly 100 tokens a second single-stream. That ceiling is why H100 looks expensive per token even at a competitive hourly rate. The GPU isn't the bottleneck. Moving data through memory is.

◆ BATCH SIZE IS THE BIGGEST LEVER

75% cost reduction from software, not hardware

Batch size is the biggest lever on H100 cost per token. Single-stream leaves most of the GPU's compute idle while it waits on memory. Batching lets that same bandwidth serve more tokens per second without adding hardware. Batch=1 to batch=8 on Llama 4 70B cuts cost per token roughly 75%. Software configuration, not a faster chip.

FP8 does something similar from a different angle. On H100, FP8 roughly doubles throughput without changing GPU count. A team running FP16 and comparing cost per token against a competitor's FP8 numbers is comparing two operating points on the same hardware, not two different GPUs.

◆ WHAT "H100 RATE" ACTUALLY MEANS

Source: GPU Tracker 2026 Cloud GPU Statistics, 5,213 live listings across 54 providers

◆ THE GAP BEHIND THE SECONDARY MARKET DROP

Not a glitch, a real economic shift

Here's the part that matters for anyone deciding whether H100 makes sense at all right now. It costs roughly 11 times more to run inference on an H100 than on a B300, per CloudZero's analysis. Genuinely large gap. Not marketing exaggeration. Direct consequence of Blackwell's native FP4 support and higher memory bandwidth doing structurally more work per dollar on large models. Anyone still running production inference on H100 for models that fit comfortably on newer hardware needs to charge more per token than a competitor who migrated, or accept a thinner margin.

That gap is why H100 secondary market prices moved this fast. Cards that sold for $40,000 in late 2023 traded at $21,000 to $34,000 by Q2 2026. Forecasts point to another 10 to 20% decline as more H100 inventory returns to the secondary market from enterprises finishing multi-year contracts and migrating to B200. Not a market glitch. Blackwell making Hopper-class hardware economically obsolete for a specific slice of workloads, mostly inference on large models, while H100 stays genuinely competitive everywhere else.

◆ WHERE H100 STILL WINS, AND WHERE IT STOPS

The 13B-70B lane, until memory becomes the constraint

H100 still wins a specific lane even with B200 and H200 fully available. For throughput-sensitive production APIs serving 13B to 70B parameter models, H100 edges out A100 and, in a meaningful share of FP8 configurations, holds its own against newer tiers too. Not a nostalgia argument. A real result tied to how well H100's Transformer Engine handles quantized inference at that model size range. It's the reason more than one market analysis still calls H100 "broadly rentable and often the value choice," not a chip anyone should be embarrassed to run.

H100 stops winning exactly where memory bandwidth becomes the binding constraint rather than compute. A model needing more than 80GB to serve in FP16 forces quantization tradeoffs or a second GPU, either of which changes the cost math H100 was supposed to deliver. H200's 141GB fits many of those same models on one card, at a rate that has to be checked against what the second H100 would have cost, since the honest comparison is never one H100 against one H200.

◆ FAQ

Frequently asked questions