Nobody Tells You How the GPU Market Actually Works

◆ THE PRICING PAGE PROBLEM

What you are actually buying when you rent a GPU

When AWS lists the p5.48xlarge at $55.04 per hour, that rate covers eight H100 SXM5 GPUs. Divide by eight and you get $6.88 per GPU per hour. That is the number that goes into the procurement model.

What the $55.04 actually includes: eight H100 GPUs, 192 vCPUs, 2TB of RAM, 30TB of NVMe storage, and an elastic networking fabric. If you are running LLM inference, the GPUs are saturated and most of the rest sits idle. You pay for all of it.

Then come the charges that do not appear on the pricing page at all. Egress is $0.08 to $0.12 per GB on most hyperscalers. Moving a 100GB model checkpoint out of AWS is $8 to $12 on top of compute. Teams doing regular checkpoint syncs or serving across regions can end up with egress charges that rival their GPU bill. Persistent storage is billed separately at $0.10 to $0.30 per GB per month. Running in Europe or Asia adds a 10 to 30% regional premium.

Cast AI tracks GPU pricing across providers using live API feeds and telemetry from millions of scheduling events. Their read on the market: GPU prices follow hype and scarcity, not logic. The number on the pricing page and the number on the invoice are rarely the same.

The spread on H100 pricing in 2026 is 4.7x across provider types: $1.49 per hour on spot marketplace instances, $6.98 on Azure on-demand. Same GPU. The difference is how the hardware is bundled, where it sits in the supply chain, and what kind of contract you are on.

◆ HOW GPU INVENTORY ACTUALLY WORKS

The on-demand pool is not the whole market

Most buyers assume GPU capacity works like general cloud compute: there is a pool, you request from the pool, you get capacity or you wait a bit. That model stopped being accurate sometime in 2024.

The bulk of H100, H200, and B200 inventory gets sold on 1 to 3 year reserved contracts before it is deployed, sometimes before the hardware ships. Cast AI's GPU price report says it directly: most H100s and B200s are already pre-sold, tied to financial commitments spanning one to three years. What reaches the on-demand pool is whatever those contracts did not absorb.

This has a few practical effects. On-demand inventory is smaller and less predictable than buyers expect. Spot pricing spikes during demand windows because spot draws from the same constrained pool. And availability varies dramatically by region. AWS might show H100 capacity in us-east-1 with nothing in us-west-2. Cast AI found a 45% availability gap between two GCP availability zones in the same region in June 2025.

If you are trying to provision GPU capacity through a standard on-demand channel for the first time, you are competing for residual inventory. The main market cleared before you got there.

GPUaaS.com works differently. Buyers submit requirements and receive quotes from vetted providers within 24 hours, including clusters that are not publicly listed anywhere. That visibility into pre-market inventory is worth more in 2026 than it was two years ago.

◆ SPOT PRICING

When spot works and when it doesn't

The AWS H100 spot price dropped 88% between January 2024 and September 2025: from $105.20 to $12.16 for an 8-GPU instance. That is an 8.65x cost efficiency gain in 20 months. For batch training jobs with checkpointing, spot is genuinely the cheapest path to GPU compute available.

The problem is using spot as a planning input. Cast AI tracks this: spot pricing for GPU instances has become unreliable for teams trying to build predictable cost models. Enterprises that ran development workloads on spot in 2024 have found the same instances unavailable at any price during high-demand windows in 2026.

Spot markets exist for supply-demand balancing, not cost stability. A cluster priced at $12.16 in September 2025 may simply not appear during a demand spike. Teams that built workflows around those rates and then hit a demand window had to pay on-demand prices or wait. Neither option works when something is in production.

Spot is the right choice for fault-tolerant batch training, offline inference, and anything that checkpoints and resumes. It is not a reliable foundation for production inference or any workload with a fixed deadline. For the decision framework, the reserved vs on-demand GPU guide covers it in full.

◆ WHERE THE BUDGET GOES

Inference ate the training budget and nobody noticed until the invoice arrived

Two years ago AI infrastructure spend was training-heavy. Big model runs were expensive, visible, and approved upfront. Inference costs were smaller and more predictable.

That has flipped. Inference now accounts for roughly 80% of AI infrastructure budgets according to GPUnex 2026 analysis, up from a training-dominated split just two years earlier. Per-token inference cost has fallen about 1,000x in three years, from around $20 per million tokens in early 2024 to under $0.40 by early 2026. Total spend keeps climbing because usage grows faster than unit cost falls.

Finance teams get surprised by this because inference spend does not arrive as one big line item. It accumulates as more AI features ship, as more users interact with products, as agent workflows start making more API calls per task. By the time anyone notices the GPU bill has tripled, the workload driving it has been running for months.

Cast AI 2026 data shows average GPU utilization across 23,000 production clusters at 5%. At that utilization rate, a $6.88/GPU/hour instance has an effective cost of $137.60 per hour of actual compute work. A cluster running at 20% utilization is paying 5x the real compute cost. Most teams have no visibility into their actual utilization until they specifically go looking for it.

For a breakdown of what GPU billing structures mean for inference-heavy workloads, the GPU billing models guide for finance teams covers the anatomy in detail. The GPU bill spike explainer covers why bills jump unexpectedly and what to do about it.

◆ RESERVED CONTRACTS

Committing early is not always the winning move

There is a common assumption that teams who locked in GPU capacity early got the better deal. Sometimes that is true. AWS cut H100 pricing by roughly 44% in mid-2025, one of the sharpest GPU price drops in cloud history. Teams that had signed 1 or 3-year reserved contracts at 2024 rates ended up paying more than new on-demand customers for exactly the same hardware. The commitment that felt like cost certainty became a cost liability when the market moved.

This is not an argument against reserved contracts. For stable inference workloads with predictable demand, 12 to 24 month commitments still save 30 to 40% over on-demand rates. The point is that a reserved contract is a bet on price stability. In a market where a single pricing action can move rates 44%, that bet needs more analysis than most teams apply before signing.

The teams navigating this well in 2026 match commitment length to how confident they are about their workload demand. Short contracts while a workload is still being validated. Longer commitments once inference demand has stabilized and the pattern is clear. And provider relationships across more than one source, so when one rate becomes uncompetitive there is somewhere to go.

GPUaaS.com does not require multi-year commitments upfront. Pricing reflects what the market actually is rather than a hyperscaler retail markup. For the current rate picture across GPU models, the GPU pricing guide has the numbers. For H100 specifically, the H100 cost per hour analysis goes deeper. And for how wholesale rates compare to hyperscaler pricing, the wholesale vs hyperscale breakdown explains the gap.

◆ WHAT TO ACTUALLY DO

Four things that change your GPU procurement outcomes

None of this is especially hard once you know it. The problem is most buyers do not know it until after the first contract.

Normalize to per-GPU-hour before comparing anything. Hyperscalers quote per instance. Purpose-built providers quote per GPU. Google Cloud quotes per VM. These are not the same number and comparing them directly will mislead you. The 4.7x H100 pricing spread is invisible until you put everything on the same basis.

Model egress and storage before you sign. Take your expected monthly data transfer volume, multiply by $0.08 to $0.12 per GB. Add persistent storage at your dataset and checkpoint sizes. Apply any regional multiplier. This usually adds 20 to 40% to the headline rate. Do it before the contract, not after the invoice.

Match your commitment length to your utilization certainty. If you know what your inference workload looks like in 18 months, commit for 18 months. If you are still in validation, sign something short and extend when the pattern is clear. The 40% discount on a 3-year term is only worth it if the workload is actually stable for 3 years.

Get visibility into inventory that has not been publicly listed yet. On-demand pools are the overflow after reserved contracts have been filled. GPUaaS.com surfaces clusters from vetted providers before they hit public listing, including capacity not visible through hyperscaler portals. Buyers who submit requirements receive quotes within 24 hours. That window is often the difference between securing a cluster this month and joining a waitlist.

◆ FAQ

Frequently asked questions