Cost per token is the number that actually determines whether an inference workload is economical, and it depends on far more than the GPU's hourly rental rate. The same H100 running the same model can produce wildly different cost-per-token figures depending on the serving engine (vLLM, SGLang, TensorRT-LLM all perform differently), the batch size (batching multiple requests together can cut cost per token by 3-4x versus single-stream serving), and the quantization used (FP8 roughly doubles throughput over BF16 on H100, directly halving cost per token for the same workload). A real example: H100 serving Llama 4 70B via vLLM runs about $0.45 per million tokens single-stream, dropping to roughly $0.12 per million tokens batched at 8 concurrent requests, a nearly 4x reduction from batching alone with no change in hardware or hourly rate. H200 cost per token can be lower in total when H200's memory avoids a multi-GPU H100 split.
Converting an hourly GPU rate into a cost-per-token figure requires knowing the actual throughput your specific setup achieves, which is where most naive cost estimates go wrong. Two teams running the identical H100 at the identical hourly rate can end up with cost-per-token figures that differ by 4x or more, purely based on serving engine choice and batch size. SGLang's RadixAttention, for instance, gives real throughput gains specifically when requests share common prefixes, which matters enormously for chat applications with repeated system prompts but less for one-off completions. FP8 quantization is close to a free throughput doubling on H100 since it was the first GPU generation to support it natively, meaning a workload still running BF16 is very likely leaving meaningful cost savings on the table without any hardware change at all.
The hourly rate you pay for H100 capacity is one input into cost per token, and it varies meaningfully by region and provider. See H100 availability by country below.
Read the full guide to GPU cloud in this location →Wholesale rates against cloud list price for a 64-GPU cluster.
We connect you to our vetted partners. You contract directly with the operator running your nodes.
GPU model, count, placement and timeline. Add workload detail if you have it.
We find vetted partners with capacity that fits, in the jurisdiction you need.
Real quotes from partners who hold the capacity, not listings that may not exist.
You contract directly with the operator. We smooth the provisioning process.
Tell us the essentials. We'll line up real quotes from our vetted wholesale partners, and you contract directly with the operator.
GPUaaS.com connects enterprises with vetted GPU providers for large-scale compute at up to 30% lower cost than hyperscalers.
