The H200 and H100 share the same Hopper compute engine — same Tensor Cores, same FP8 Transformer Engine, identical FLOPS. The only meaningful difference is memory: the H200 carries 141 GB HBM3e at 4.8 TB/s versus the H100's 80 GB HBM3 at 3.35 TB/s. That single upgrade determines everything about which one you should rent.
- H200 and H100 have identical compute specs. The H200's advantage is 76% more VRAM (141 GB vs 80 GB) and 43% more bandwidth (4.8 TB/s vs 3.35 TB/s)
- In MLPerf Llama 2 70B inference, H200 achieves 31,712 tokens/sec vs H100's 21,806 — a 45% throughput advantage on that memory-bound workload (NVIDIA)
- For sub-70B models where weights fit comfortably in 80 GB, H100 and H200 perform nearly identically. H100 is cheaper and the right call
- H200 on GPUaaS.com starts at $3.50/GPU/hr on-demand. H100 starts at $1.49/GPU/hr — the H200 premium is roughly 2.3x, which the throughput gain on large models more than justifies
- The decision rule: if you're serving 70B+ parameter models, running 32K+ context windows, or your H100 setup requires multi-GPU sharding just to fit model weights, rent the H200
Most teams renting GPUs in 2026 frame this as a newer-is-better question and reach for the H200 by default. That's often the wrong call. If your model fits in 80 GB and your context windows stay short, you're paying a 2.3x hourly premium for headroom you don't use. This guide gives you the framework to pick correctly for your actual workload.
For VRAM sizing on any model, see the VRAM guide. For cluster options on both GPUs, see the GPUaaS.com cluster catalogue.
The H200 isn't a new architecture. It's the same GH100 Hopper die as the H100, with the memory subsystem replaced. Everything compute-related — CUDA cores, Tensor Core generation, FP8 support, NVLink 4.0 bandwidth — is identical. If your workload is compute-bound rather than memory-bound, you won't see a difference.
| Spec | H100 SXM5 | H200 SXM | Impact |
|---|---|---|---|
| Architecture | Hopper (GH100) | Hopper (GH100) | Identical — same die |
| VRAM | 80 GB HBM3 | 141 GB HBM3e | +76% — the core upgrade |
| Memory bandwidth | 3.35 TB/s | 4.8 TB/s | +43% — faster inference decode |
| FP8 TFLOPS | 1,979 | 1,979 | Identical |
| BF16 TFLOPS | 989 | 989 | Identical |
| NVLink bandwidth | 900 GB/s | 900 GB/s | Identical |
| TDP | 700W | 700W | Identical |
| Price/GPU/hr (GPUaaS.com) | from $1.49 | from $3.50 | +2.3x — justifiable for large models |
The one-line summary
If your job hits a memory wall on H100, the H200 fixes it. If it doesn't, you're paying 2.3x for nothing. The entire decision reduces to whether memory capacity or bandwidth is your actual constraint.
LLM inference is memory-bandwidth bound, not compute-bound. The model's attention mechanism has to read every cached key and value vector on every decode step — that's a memory operation, not a matrix multiply. Higher bandwidth means faster reads, which means more tokens per second.
The H200's 43% bandwidth advantage shows up directly in throughput, but the size of the gain depends entirely on whether the model fits in 80 GB. Once you're within 80 GB, the H100 does the same work — the extra bandwidth in the H200 isn't helping with anything.
Where H200 wins clearly
- 70B+ models at full BF16/FP16 precision
- 128K+ context windows (KV cache overflow on H100)
- High-concurrency serving where batches saturate 80 GB
- Serving multiple large models on one node
- Long-context RAG and document processing
Where H100 matches or beats H200
- Sub-70B models at FP8 or INT8 (fit in 80 GB)
- Short-context inference (4K–16K tokens)
- Compute-bound training where memory isn't the cap
- Fine-tuning runs with QLoRA (adapter is small)
- Cost-sensitive batch jobs without latency SLAs
MLPerf Llama 2 70B inference — tokens per second (offline scenario)
Source: MLPerf Inference v4.0 · Single-node SXM configs · Llama 2 70B · Offline scenario
⚡ Read the benchmark context carefully
MLPerf's 45% throughput gain applies specifically to Llama 2 70B, which saturates the H100's 80 GB in BF16. For smaller models that fit in 80 GB, independent benchmarks show gains in the low single digits — closer to 5–15%. Your actual workload matters more than the headline number.
On MLPerf Inference v4.0, H200 SXM achieves 31,712 tokens per second on Llama 2 70B in the offline scenario, compared to 21,806 for H100 SXM — a 45% throughput advantage that only materialises when the 70B model saturates H100's 80 GB. Source: Tom's Hardware.
Hourly rate comparisons between H100 and H200 are misleading in isolation. What actually matters is cost per million output tokens — and that calculation flips in favour of the H200 once you're running models large enough to saturate H100 memory.
The cost-per-token flip
For Llama 2 70B at BF16: H100 delivers ~21,806 tok/s at $1.49/hr = $0.019 per 1,000 tokens. H200 delivers ~31,712 tok/s at $3.50/hr = $0.031 per 1,000 tokens. H100 wins on raw cost-per-token for this model — unless you need the extra headroom to avoid multi-GPU setups. Once 2x H100 ($2.98/hr) becomes necessary to fit the model, a single H200 at $3.50/hr is cheaper and simpler.
GPUaaS.com data: a single H200 SXM at $3.50/hr replaces a 2x H100 setup ($2.98/hr) for teams running Llama 3 70B at BF16 with context windows above 32K — eliminating multi-GPU NVLink overhead and reducing operational complexity.
Run through these four questions in order. The first one that produces a definitive answer is your decision.
Do your model weights exceed 70 GB at your serving precision?
Yes → Rent H200. The model won't fit on H100 without quantisation or sharding. No → Continue to Q2.
Are you serving at context lengths above 32K tokens?
Yes → Likely H200. At 32K+ context, the KV cache starts competing with weight memory on an H100 — see the KV cache guide for exact figures. No → Continue to Q3.
Does your current H100 setup require multiple GPUs just to fit model weights?
Yes → Run the maths: 2x H100 at $2.98/hr vs 1x H200 at $3.50/hr. If a single H200 covers your capacity requirements, it's simpler and likely cheaper. No → Continue to Q4.
Is your workload hitting throughput limits on H100 that you'd attribute to memory bandwidth?
Yes → H200's 4.8 TB/s vs 3.35 TB/s bandwidth gap may help. Benchmark on both before committing. No → Rent H100. You're paying 2.3x for memory you don't need.
Quick reference by workload type
| Workload | Recommended | Why |
|---|---|---|
| Llama 3 8B / Mistral 7B inference | H100 | Fits in 80 GB at FP16; H200 bandwidth gain is minimal |
| Llama 3 70B at FP8, short context | H100 | FP8 weights ~70 GB, fits in 80 GB with room |
| Llama 3 70B at BF16, 4K–16K context | H200 | 140 GB weights fill H100; one H200 replaces two H100s |
| Llama 3 70B, 128K context window | H200 | KV cache at 128K adds ~42 GB on top of weights |
| QLoRA fine-tuning, 13B–70B | H100 | 4-bit base + adapters fit comfortably in 80 GB |
| Full fine-tuning, 70B+ | H200 | Weights + gradients + optimizer states need 400 GB+ total |
| DeepSeek-V3 / R1 inference | H100 | MLA architecture cuts KV cache by 93%; memory pressure is low |
| Multi-turn RAG, 64K+ context | H200 | Context-length KV cache makes H100 OOM at scale |
The single strongest argument for the H200 isn't throughput on standard benchmarks — it's what happens to your infrastructure when you need 128K+ context windows on a 70B model. At 128K context, the KV cache for Llama 3 70B at BF16 adds approximately 42.9 GB on top of the 140 GB weight footprint. That's 183 GB total — two H100s minimum.
One H200 at $3.50/hr handles this with FP8 KV quantisation enabled. Two H100s at $1.49/hr each cost $2.98/hr, require NVLink coordination, and add latency from cross-GPU communication. The H200 is simpler, faster at serving, and only 17% more expensive per hour.
The NVLink tax
Tensor parallelism across multiple H100s adds NVLink communication overhead — every all-reduce operation during decode requires synchronisation across GPUs. For inference (not training), this overhead often negates the raw throughput gain of having two GPUs. A single larger-memory GPU is almost always faster for serving than two smaller ones joined at the hip.
The exception worth knowing: DeepSeek-V3 and R1 use MLA (Multi-head Latent Attention), which compresses the KV cache by 93%. If you're serving DeepSeek models, your memory pressure is radically lower and an H100 stays viable at much longer context lengths. For a detailed breakdown of how KV cache size scales with context, see the KV cache inference cost guide.
⚠ One exception: 8xH100 clusters
For very large model training (405B+) or massive distributed inference, 8xH100 SXM NVLink clusters (640 GB total) can be more cost-effective than 4xH200 (564 GB) for compute-bound workloads. At that scale, the H100's lower hourly rate compounds significantly. Check 8xH100 cluster pricing on GPUaaS.com before assuming H200 is the answer.
GPUaaS.com infrastructure data: teams migrating from 2xH100 to 1xH200 for 70B BF16 inference at 32K+ context report a 17% reduction in hourly cluster cost and elimination of tensor-parallelism latency overhead.
Both GPUs are on the same Hopper software stack. Your container, your vLLM or TGI config, your model weights — all of them migrate without changes. There's no driver update, no recompilation, no code change. The only thing that changes is your --gpu-memory-utilization flag and whatever KV cache settings you've tuned for H100's 80 GB.
H100 in 2026
Broad availability across US-East, US-West, EU-West, and Southeast Asia. H100 pricing has dropped steadily since mid-2024 as Blackwell supply increased pressure on the market. On-demand rates on GPUaaS.com start at $1.49/GPU/hr — roughly 40% below early-2024 peaks.
H200 in 2026
Availability has improved significantly from early 2025. GPUaaS.com carries H200 SXM clusters in US-East and EU-West at $3.50/GPU/hr on-demand. Spot pricing and reserved commitments are available for teams with predictable workloads — contact the team via how it works for volume rates.
Migration checklist
Moving from H100 to H200 (or back): update --gpu-memory-utilization to reflect the new VRAM budget; if you had FP8 KV cache tuned for 80 GB, re-tune for 141 GB; re-run your latency and throughput benchmarks at your production batch size and context length. Everything else — model weights, inference engine, code — is identical.
Last reviewed: May 27, 2026. For live H100 and H200 cluster pricing, visit the H100 cluster page or H200 cluster page on GPUaaS.com.
