Storage and Data Pipeline Costs Nobody Budgets For

◆ THE SAME BYTES, TWENTY TIMES THE PRICE

Sources: Spendark ML cloud cost analysis (June 2026), OneSource Cloud AI infrastructure pricing (June 2026)

◆ THE CHEAPEST TIER IS WHERE MONEY IS LOST

22% GPU idle against 5%

The spread between tiers is what makes this easy to get wrong. That is a factor of more than twenty between the cheapest and most expensive option for storing the same bytes.

One benchmark across three storage configurations on a 30-day training run of a 10B model measured GPU idle time directly. Local NVMe at 25 Gbps throughput produced 5% GPU idle. NFS over SSD at 4 Gbps, costing $3,200 for the run, produced 22% GPU idle.

Seventeen percentage points of idle time on a cluster of any meaningful size costs far more than $3,200. The storage saving is real and the GPU waste is larger. This is the same mistake as buying a cheaper fabric and discovering the GPUs are waiting on data, and it shows up in exactly the same place: unexplained low utilisation that looks like a software problem.

◆ THE TIERING RULE

Mixing them up costs money in both directions

Local NVMe for training data. Parallel file system for shared workloads. Object storage for cold artifacts. Mixing those up costs money in both directions, either by paying parallel-file-system rates for checkpoints nobody reads or by starving GPUs to save on object storage.

A working target for local NVMe is 5 GB/s or better in read throughput per node, with roughly a 1:1 GPU-to-storage ratio. Below that, GPUs start waiting.

The parallel file system itself is a capital line most budgets miss entirely. A WEKA node with around 50TB of NVMe runs $60,000 to $100,000, and four to eight nodes is typical for a medium cluster. That is $240,000 to $800,000 before a single GPU is counted.

◆ CHECKPOINTING IS A TRAINING-TIME QUESTION

Storage cost stops being about storage

A 70B model checkpoint is roughly 140GB in FP16, and considerably larger once optimizer state is included. Saving hourly produces several terabytes a day. On object storage at $0.023 per GB per month that is genuinely negligible. On local NVMe it is roughly $2,000 of SSDs per node.

The number that matters more is recovery. If a crash forces a restore and storage takes 45 minutes to push a checkpoint back to 512 GPUs, that single failure costs $5,000 to $8,000 in burned compute. At the failure rates that come with large clusters, that arithmetic compounds quickly, and it is why checkpoint read bandwidth is worth specifying rather than assuming.

This connects to something measured on a production 504-GPU cluster: checkpoint writes spent 93.1% of their request time queued rather than writing. The storage layer is the bottleneck, not the write itself, and no amount of checkpoint frequency tuning fixes a queue. The failure rates behind that arithmetic are covered in multi-node training at 512 GPUs and above.

◆ EGRESS SURPRISES FINANCE, NOT ENGINEERING

Up to 30% of total infrastructure cost

Public cloud egress runs $0.05 to $0.12 per GB, with AWS at $0.09 for internet egress from us-east-1. For a single 8-GPU cluster with moderate data movement, that is $31,000 to $43,000 annually. For data-intensive workloads, egress can reach 30% of total AI infrastructure cost.

Cross-facility movement is worse because it is invisible until the invoice arrives. If storage and compute sit in different facilities or network segments, transfers accumulate at $0.08 to $0.12 per GB. A pipeline reading 10TB per epoch across 50 training runs generates $40,000 to $60,000 in data movement charges annually, for data that never left the organisation.

◆ THE LINE THAT BILLS WHEN NOTHING RUNS

GPUs stop. Storage does not.

Persistent storage is the part that keeps costing money when nothing is running. A team managing 50TB of active training data and 200TB of model artifacts faces $5,000 to $25,000 monthly depending on tier. GPUs stop billing when the job ends. Storage does not.

Tiering is the obvious fix and the commonly skipped one. Keeping cold artifacts on hot tiers is a compounding error, growing with every completed experiment that nobody deletes. Most teams have more old checkpoints than they realise, and almost none of them are ever read again.

◆ THREE QUESTIONS THAT ARE NOT ON A RATE CARD

Ask before signing

What is the measured read throughput per node, and does it clear 5 GB/s. What does checkpoint restore look like at the cluster size being quoted, measured rather than estimated. And is storage billed separately from compute, including egress between the storage tier and the GPUs.

That last one catches people. Storage billed separately is normal and fine. Storage billed separately with egress charges between tiers inside the same provider is a different proposition, and it is worth reading the terms rather than assuming.

The underlying point is that a GPU quote is not an infrastructure quote. Compute is the line everyone negotiates and storage is the line that quietly determines whether the compute is doing anything. On data-heavy workloads the storage bill can exceed the GPU bill outright, which makes it worth modelling before the cluster is sized rather than after the first invoice.

For the related question of how fabric choice affects whether GPUs wait on data, see InfiniBand vs Ethernet in 2026.

◆ FAQ

Frequently asked questions