Batch Size, Quantization, Throughput: Three Levers That Beat New Hardware

◆ WHAT EACH LEVER DELIVERS

Sources: LLM-Inference-Bench (Llama 3 8B, vLLM, single A100), QQQ quantization benchmarks on Llama 2 13B, MyEngineeringPath inference optimization analysis, March 2026

◆ WHY BATCHING IS THE LARGEST LEVER

Fixed costs happen once per batch, not once per request

Throughput rises with batch size until compute and memory saturate. The GPU processes more sequences at once, and fixed costs like kernel launch happen once per batch rather than once per request. Continuous batching, default in vLLM and TensorRT-LLM, admits new requests as slots free rather than waiting for a batch to drain. Static batching leaves the device idle whenever one sequence in a batch finishes early and the rest are still generating.

What caps batch size is memory, not compute. At 2048-token sequence length, KV cache growth can limit a batch to around seven sequences. Every byte freed elsewhere becomes batch capacity.

◆ QUANTIZATION RAISES THE BATCHING CEILING

The two levers multiply rather than add

INT8 quantization halves memory at typically under 1% quality loss. INT4 through AWQ or GPTQ cuts memory 75%, enough to move a 70B model from two GPUs to one. On a serving deployment that is a 50 to 75% reduction in GPU spend.

Throughput improves too. Benchmarked on Llama 2 13B, a well-implemented 4-bit scheme runs up to 2.24 times faster than FP16, 2.10 times faster than SmoothQuant, and 1.59 times faster than AWQ. The advantage grows with batch size rather than shrinking, and larger models benefit more than smaller ones.

The two levers multiply rather than add. Halving memory doubles the space available for batching, so quantization raises the ceiling on the first lever while lowering per-token cost. Teams applying both see gains well beyond either alone.

◆ THE ONE THAT COSTS NOTHING IN QUALITY

Speculative decoding, and where it collapses

Speculative decoding is mathematically free. A small draft model proposes tokens, the large model verifies them in one forward pass, and verification costs less than generation. Speedup runs 1.5 to 3 times and the output is identical to standard decoding.

The gain depends on output predictability. Structured data and templated responses give the draft model a high hit rate. Creative writing drives it down and the speedup approaches zero. Worth testing on production traffic rather than assuming.

PagedAttention and prefix caching deliver 2 to 4 times throughput and take a day or two to configure. Most modern serving stacks enable them by default, which is worth verifying rather than assuming.

◆ THE CHEAPEST LEVER OF ALL

Prompt length is billed on every request

Prompt optimization sits below all of these in effort and above most in return. Shorter system prompts reduce prefill cost on every single request, and prefix caching makes a shared system prompt close to free after the first call. Teams running a 2,000-token system prompt on every request are paying for it on every request.

◆ ORDER OF OPERATIONS

Applying the wrong technique produces effort without improvement

Cost, latency, throughput and quality are different problems with different fixes. Identify which one binds, apply the lowest-effort intervention that addresses it, measure again on the same evaluation suite, roll back if quality dropped.

◆ WHERE DISAPPOINTING RESULTS COME FROM

Quantization failure is silent

Quantizing without benchmarking on your own data is where most disappointing results come from. MMLU and HumanEval scores do not reflect a specific production workload. A model that holds up on academic benchmarks can degrade badly on domain-specific queries after quantization. Mathematical reasoning is particularly sensitive because quantization loses numerical precision directly. Models below 13B suffer proportionally more, having less redundancy to absorb the loss.

Quality evaluation needs a fixed test set drawn from real traffic before any of this starts. Without a baseline measured on the same set, there is no way to tell whether a 4-bit conversion cost anything, and the failure mode is silent. Nothing errors. Answers just get slightly worse in ways that surface as user complaints weeks later.

Traffic patterns shift and techniques mature. A quarterly revisit is enough.

◆ THE PROCUREMENT CONSEQUENCE

Buying more of the same configuration buys more waste

Before signing for more GPUs, the question worth answering is whether the current fleet runs at default batch settings in FP16. If it does, the available software gains likely exceed what the expansion delivers, at a fraction of the cost. A 26.6x throughput difference from a configuration value is larger than any generational hardware jump on offer.

This is the same pattern as a dashboard reporting 100% utilization on a fleet doing very little useful work. Capacity looks exhausted because the configuration is wasting it, and buying more of the same configuration buys more waste.

There is a provider-side version of this question too. A quote priced against default-configuration throughput assumes a number of GPUs that an optimized stack would not need. Running the optimization work before requesting the quote changes what gets ordered, not just what it costs.

None of this argues against upgrading hardware. It argues for exhausting the free levers first, because the measurement that tells you whether more GPUs will help is the same one that tells you whether the current ones are configured properly. For how that measurement changes cluster sizing, see B300 cluster sizing.

◆ FAQ

Frequently asked questions