Blog ▸ Long Context Is Expensive: What 128K Windows Do to Your Bill
GPU Infrastructure
A 128K context needs eight times the KV cache storage and sixty-four times the attention computation of 16K. Memory scales linearly, compute quadratically.
Long Context Is Expensive: What 128K Windows Do to Your Bill
GPUaaS.com Team
GPU Infrastructure
September 16, 2026
No items found.
A 128K context is not eight times more expensive than 16K.
It needs eight times the KV cache storage, and under full attention it needs sixty-four times the attention computation.
Key takeaways
Memory grows linearly with context and compute grows quadratically. Treating them as one number gets both wrong
A 70B model at 1M context carries roughly 135 GB of KV cache against 140 GB of weights. The cache is nearly as large as the model
At 256K tokens, prefill attention takes 57.76 seconds, which is 89.51% of time to first token
RULER found only about half of 17 tested models held satisfactory performance even at 32K, well short of advertised windows
Prefix caching at a 90% hit rate cuts compute cost per request 80 to 90%. One estimate puts it at $20,000 monthly against $2,000
◆ TWO COST CURVES, ONE INVOICE
Memory is linear. Compute is quadratic.
Those are two different cost curves. Memory grows linearly with context while compute grows quadratically, and most capacity planning treats them as one number and gets both wrong.
Start with memory, because it is the one that shows up as an out-of-memory error. For a 70B-class model at BF16 serving one user, KV cache runs about 10.7 GB at 32K context and 42.9 GB at 128K. At 512K it is 171 GB, and at 1M it reaches 343 GB.
◆ KV CACHE MEMORY BY CONTEXT LENGTH, 70B AT BF16, ONE USER
Hugging Face publishes the Llama 3 70B figure at 128K as 39.06 GB. The formula behind it is twice the product of layer count, KV head count, head dimension, sequence length, batch size, and bytes per parameter.
Three crossover points matter. Above 32K, KV memory starts outpacing parameter memory, and above 128K it dominates outright. At 1M tokens it consumes 70 to 90% of GPU VRAM and 60 to 85% of wall-clock time per token.
On a Llama 70B at 1M context, KV cache hits roughly 135 GB at FP16 against model weights of 140 GB. The cache is nearly as large as the model it serves.
A Llama 3.1 405B at 128K needs 66 GB of KV cache for a single user at FP16. That is one user.
8x vs 64x
the storage and compute cost of moving from 16K to 128K context. Memory scales linearly, attention computation scales quadratically, and the two need different fixes
The KV Cache Wars, Agentic AI analysis, April 2026
◆ COMPUTE HURTS IN A DIFFERENT PLACE
Prefill attention at 1M tokens exceeds an hour
Compute behaves differently and hurts in a different place. Prefill is where the model reads the entire prompt at once, and attention at that stage scales quadratically with input length.
The measured numbers are severe. At 256K tokens, self-attention during prefill takes 57.76 seconds, which is 89.51% of time to first token. At 1M tokens the attention computation exceeds one hour, accounting for 98.56% of total latency.
Those are research benchmarks rather than production figures, and production is better but not by as much as you would hope. OpenAI reports TTFT rising from seconds at around 128K to roughly a minute at 1M, on an already optimized stack.
◆ THE QUALITY PROBLEM UNDER THE COST PROBLEM
Paying for context the model cannot use
There is a quality problem underneath the cost problem, and it is the part most teams never check.
The RULER benchmark tested 17 long-context models against their own advertised windows, which ranged from 32K to 1M. Only about half maintained satisfactory performance even at 32K. Models showed 15 to 64 point drops between 4K and 128K, and some degraded more than 20% going from 2K to 8K.
A KV cache sized for 128K is a real billable memory allocation whether or not the model can use that context effectively. Provisioning for the advertised number without validating the usable number means paying for capability that is not there.
Context length is a hardware spec. Not an intelligence spec. Doubling the window does not double comprehension, it dilutes attention and doubles the memory bill.
◆ THE LARGEST COST LEVER
$20,000 a month against $2,000 for the same workload
The single largest cost lever is prefix caching. The gap it opens is not marginal.
At 100:1 input to output token ratios, prefill accounts for 85 to 95% of total GPU time per request. A 90% cache hit rate skips 90% of that work, dropping TTFT from several seconds to under 200ms and cutting effective compute cost per request by 80 to 90%.
One published estimate puts the difference between a 0% and a 90% hit rate at a $20,000 monthly GPU bill against $2,000 for the same workload.
◆ WHAT ELSE MOVES THE NUMBER
Paged attention is the substrate, not an optimization
FP8 KV quantization halves every memory figure above. An H200 SXM5 with 141 GB can serve a 70B model with full KV cache at around 128K per user slot at FP8, and beyond 500K tokens NVMe offload or sequence parallelism becomes necessary.
Paged attention is not an optimization. It is the substrate. Without it, GPU memory fragments under variable-length batches and 30 to 50% of VRAM goes to waste. Every production stack in 2026 has it on by default.
Applied together, the technique families collapse long-context inference cost by four to forty times. For the broader set of software levers available before any hardware change, see three levers that beat new hardware.
◆ THE PROCUREMENT CONSEQUENCE
Context length belongs in the quote
The procurement consequence is that context length belongs in a quote alongside model size and concurrency.
A 70B model at 8K context and the same model at 128K are different hardware requirements, and sizing against the model alone misses the larger variable. Sizing against an advertised context window the model cannot actually use means buying memory to hold tokens it will not attend to properly.
Measure usable context before provisioning for advertised context. The two numbers are rarely the same. For how concurrency interacts with all of this, see how cost per token scales with concurrency.
Get quoted with context length priced in.
Alongside model size and concurrency. No buyer fees. For single GPUs, packet.ai handles self-serve access with 24/7 human support.
Against a 16K context it needs eight times the KV cache storage and, under full attention, sixty-four times the attention computation. On a 70B model at BF16, that is roughly 42.9 GB of KV cache for a single user, against about 10.7 GB at 32K.
KV memory starts outpacing parameter memory above 32K and dominates above 128K. On a Llama 70B at 1M context, KV cache reaches roughly 135 GB at FP16 against 140 GB of model weights, and consumes 70 to 90% of GPU VRAM.
Often not. The RULER benchmark tested 17 models against their own claimed windows and found only about half maintained satisfactory performance even at 32K. Models showed 15 to 64 point drops between 4K and 128K. The KV cache is billed regardless of whether the context is used effectively.
Prefix caching. At 100:1 input to output ratios, prefill is 85 to 95% of GPU time per request, so a 90% hit rate skips most of it. TTFT drops from seconds to under 200ms and compute cost per request falls 80 to 90%.
Prefill attention scales quadratically with input length. At 256K tokens it takes 57.76 seconds, which is 89.51% of TTFT. At 1M tokens it exceeds an hour in research benchmarks. Production stacks do better, with OpenAI reporting roughly a minute at 1M.
Last reviewed: 17 September 2026. KV cache memory formula and RULER benchmark findings from IntuitionLabs' KV cache memory report, September 2026. Context-length memory figures and prefix caching economics from Spheron's context engineering guide, August 2026. Crossover points and technique family gains from DigitalApplied's KV cache optimization guide, April 2026. Prefill latency measurements from the VSPrefill paper and OpenAI reporting via Compresr's long context cost guide. Storage versus compute scaling asymmetry from The KV Cache Wars, April 2026. FP8 KV cache behaviour from the vLLM project blog, April 2026. Browse current GPU cluster availability on GPUaaS.com.
