Blog ▸ GPU Selection for AI Workloads: A 2026 Guide to Matching Hardware to Use Case
GPU Infrastructure
Fine-tuning, RAG, image and video generation, and agentic systems each have their own hardware sizing rules that do not reduce to a training-versus-inference framework.
GPU Selection for AI Workloads: A 2026 Guide to Matching Hardware to Use Case
GPUaaS.com Team
GPU Infrastructure
August 12, 2026
No items found.
Full fine-tuning of a 7B model needs roughly 112GB for standard Adam-style optimization, before activations. That's past a single H100's 80GB regardless of generation. QLoRA on the same model runs under 48GB, fits on one card. The method decides the hardware, not the chip generation.
Embedding models for RAG need a fraction of what generation needs. BGE-M3 runs on CPU in most pipelines. Paying for GPU capacity on the embedding step while running generation on the same tier is common and usually unnecessary.
Key takeaways
Fine-tuning method decides hardware before GPU generation does. Full fine-tuning of a 7B model needs ~112GB, QLoRA runs under 48GB on the same model
Embedding models for RAG often run fine on CPU, making GPU spend on that step frequently unnecessary
Image and video generation scale by frame count and resolution, not parameter count. Video VRAM scales quadratically with resolution, not linearly
Agentic systems size for the loop, not the first reply: agent count times base model size, plus ~30% KV cache headroom
Native FP8 roughly doubles throughput and halves VRAM versus FP16. FP4 goes further, older architectures without native FP8 support get none of this
◆ FINE-TUNING: METHOD DECIDES HARDWARE, NOT GENERATION
Same model, wildly different requirements
Full fine-tuning of a 7B model needs roughly 112GB before activations, forcing multi-GPU regardless of generation. QLoRA fine-tuning of the identical model runs under 48GB, fitting comfortably on one card. The gap between fine-tuning methods dwarfs the gap between any two GPU generations, which makes method the first real decision, not chip choice.
◆ IMAGE GENERATION: VRAM STACKS BY COMPONENT
Base model plus every adapter loaded alongside it
Image generation VRAM stacks by component, not by parameter count alone. SDXL at BF16 on 24GB: base model plus four LoRA adapters plus two ControlNets lands just under 22GB. A fifth LoRA pushes it into out-of-memory territory. Spot H100 SXM5 pricing on that workload runs $0.021 per hundred images. The same output on H100 PCIe on-demand costs close to four times that, same GPU family, different rental structure entirely.
Video generation scales differently again. Frame count doubles temporal attention memory linearly. Resolution scales VRAM quadratically. A 14B video model needs 54GB at full precision, 6GB once quantized, a larger compression ratio than typical language model quantization.
◆ AGENTIC SYSTEMS: SIZE FOR THE LOOP
Not the first reply
Agentic systems compound memory demand across every layer at once. Five concurrent specialist agents scale VRAM roughly proportionally to agent count, since each carries its own context window and its own KV cache. RAG retrieval underneath adds a vector index, an embedding model, and retrieved chunks injected into every prompt in the loop. Practical sizing: agent count times base model size, plus roughly 30% headroom for KV cache growth. That's a different calculation than sizing a single inference request with a fixed input and output.
◆ FIVE WORKLOADS, FIVE DIFFERENT SIZING RULES
Workload
What actually decides sizing
Fine-tuning
Method (full vs. QLoRA), not GPU generation
RAG / embeddings
Often CPU-viable, far below generation tier needs
Image generation
Adapters and ControlNets stacked on the base model
Video generation
Frame count (linear) and resolution (quadratic)
Agentic / multi-agent
Agent count × model size, plus KV cache headroom
◆ LATENCY SPLITS BY USE CASE
Deciding what tier is actually worth paying for
Latency requirements split sharply by use case. A voice agent's full round trip needs under 300 to 500 milliseconds, the language model component alone under 200. A chat agent has more room, under two seconds acceptable, time-to-first-token under 500ms the number users actually notice. A background workflow job has no real-time requirement, and low-latency hardware on that job is money spent on a property it never uses.
◆ MULTIMODAL PIPELINES: ONE NAME, SEVERAL WORKLOADS
Document processing is rarely one model
A document-processing pipeline handling OCR, embedding, and reasoning together is running three distinct workloads on one piece of infrastructure, not one workload that happens to touch three model types. A vision-language model handling document understanding has a different memory profile from a speech model doing real-time transcription, which has a different profile again from an embedding model indexing the extracted text. Treating the whole pipeline as a single sizing problem, and buying one GPU tier to cover all three, usually means overpaying on at least two of the three components.
◆ WHY THE FOUR-CATEGORY MISTAKE KEEPS HAPPENING
Most guides are still built around one axis
Most GPU selection guides default to a single axis: training on one side, inference on the other, with model size as the main variable inside each. That axis made sense when most enterprise workloads genuinely were one or the other. It stops working once fine-tuning, RAG, image and video generation, and agentic systems are all running in production side by side, each with a sizing rule that has almost nothing to do with the other four.
A single enterprise AI stack in 2026 commonly runs all five patterns at once: a fine-tuned model for domain-specific behavior, a RAG layer for grounding, an image or document pipeline for multimodal input, and an agentic orchestration layer coordinating all of it. Provisioning one GPU tier for the whole stack, sized against whichever workload looks heaviest on paper, routinely overpays for the lighter workloads and still underprovisions for the heaviest one.
~4x
the throughput gain from native FP4 support versus FP16, at roughly a quarter of the VRAM footprint, on hardware that supports it
Spheron 2026 GPU pricing and inference benchmarks
Precision format is the lever that touches nearly every category above. Native FP8 roughly doubles throughput and halves VRAM against FP16. FP4 goes further, close to 4x throughput at a quarter the memory. Older architectures without native FP8 support don't get any of this. The gap widens with every additional inference call a task makes, which is exactly the pattern agentic and multimodal pipelines produce.
None of this reduces to training versus inference. Fine-tuning method decides hardware before generation does. Embeddings need a fraction of what generation needs. Image and video scale by frame count and resolution, not parameter count. Agentic systems size for the loop, not the first reply.
Get matched to the tier that fits the actual workload.
Fine-tuning, RAG, multimodal, or agentic, not a generic training-or-inference guess. No buyer fees. For single GPUs, packet.ai handles self-serve access with 24/7 human support.
No. The gap between full fine-tuning (~112GB for a 7B model) and QLoRA (under 48GB for the same model) is far larger than the gap between any two GPU generations. Method should be decided before GPU tier is chosen.
Often not. Models like BGE-M3 run fine on CPU in many RAG pipelines. Running embeddings on the same GPU tier used for generation is a common, usually unnecessary cost.
Frame count scales temporal attention memory linearly, and resolution scales VRAM quadratically. A 14B video model can need 54GB at full precision versus 6GB once quantized, a much larger compression ratio than typical language model quantization.
Size VRAM for agent count multiplied by base model size, plus roughly 30% headroom for KV cache growth across the loop. This differs from sizing a single inference request, since agentic tasks accumulate context across many sequential calls.
Submit a workload spec, fine-tuning method, model size, agent count, latency requirement, and GPUaaS returns a competitive quote for a tier that fits the actual shape of the workload within 24 hours.
Last reviewed: 13 August 2026. Fine-tuning VRAM data from SitePoint's Fine-Tune Local LLMs 2026 guide and RunPod's Best GPU for AI Training and Fine-Tuning guide. Agentic sizing from VRLA Tech's Best Workstation for Agentic AI 2026 and GPU Mart's GPU Requirements for AI Agents 2026 guide. Image and video generation VRAM data from Spheron's Best GPU for AI Image Generation 2026 guide and Will It Run AI's Video Generation VRAM Requirements 2026 guide. Latency benchmarks from Spheron's GPU Infrastructure for AI Agents playbook. Browse current GPU cluster availability on GPUaaS.com.
