20 Neocloud storage is the part of an AI cloud that quietly determines whether the expensive part, the GPUs, earns its keep. A neocloud rents GPU capacity for training and inference, and its whole economic model rests on keeping those GPUs busy. The uncomfortable truth is that idle GPUs are usually a storage problem, not a compute problem: the chips are waiting for data that is not arriving fast enough. That makes the storage layer a first-class design decision for anyone building or supplying a GPU cloud, not a box to tick after the compute is chosen. This post explains why storage governs utilization and what properties a neocloud’s storage actually needs. If the neocloud category itself is new to you, what a neocloud is gives the background. This post assumes that context and goes straight at the storage. Why storage decides GPU utilization A modern GPU can process training data extraordinarily fast, which means it is only as productive as the pipeline feeding it. When many GPUs train in parallel, they collectively demand enormous, sustained read throughput, and any part of the path that cannot keep up leaves silicon idle. Because current-generation accelerators (NVIDIA H100 and H200, and the newer B200 and GB200 parts, as of 2026) represent a large capital and rental cost, even a few percentage points of idle time is expensive across a cluster. The result is a simple design rule for neoclouds: size storage for throughput to keep compute busy, not merely for capacity to hold the data. This is why the storage conversation for AI clouds is really a throughput conversation. The tradeoffs between capacity, bandwidth, and concurrency are laid out in object storage throughput. Where storage fits in the AI cloud stack A neocloud typically layers its storage. A durable, scalable tier holds the raw datasets, the processed training data, and the model checkpoints and artifacts, often built on object storage because it scales into petabytes and exposes a standard interface. In front of that, a faster tier (a parallel file system or a flash cache) stages the data GPUs are actively reading so the durable tier does not have to serve every random access at full speed. The art is in moving data between these tiers so the fast tier stays full of what the GPUs need next while the durable tier remains the system of record. The wider architecture around this is described in the AI cloud infrastructure architecture overview and, for the facility view, what an AI data center is. The storage properties an AI cloud needs A handful of properties separate storage that keeps GPUs fed from storage that starves them. Aggregate throughput. The system has to sustain very high read bandwidth across the whole cluster, not just in a single-stream benchmark. What matters is the total the fabric can deliver when thousands of GPUs read at once. Parallel access. Distributed training reads data from many workers simultaneously, so the storage has to serve massive concurrency without a central bottleneck. Object storage and parallel file systems are designed for exactly this fan-out. Scale. Training corpora and checkpoint histories run to many petabytes, so the durable tier must scale horizontally without forcing re-architecture as datasets grow. Durability. Datasets and trained models are valuable intellectual property, so the durable tier needs strong protection (erasure coding, multi-site options) and resilience against failures and ransomware. The object-storage foundation for AI data is covered in S3 storage for AI workloads. The data path: feeding GPUs directly Beyond raw bandwidth, the path the data takes into GPU memory matters. Traditionally, data read from storage passes through a CPU bounce buffer in system memory before reaching the GPU, which adds latency and burns CPU cycles that could be doing other work. NVIDIA’s Magnum IO GPUDirect Storage addresses this by creating a direct memory-access path between storage and GPU memory, letting data skip the CPU bounce buffer. For data-hungry training, this reduces latency and frees CPU capacity, which helps keep the pipeline full. The mechanics and when the direct path helps are covered in GPU Direct Storage. Checkpoints and the write side Reads get most of the attention, but the write side can stall a run just as easily. Long training jobs periodically save checkpoints, which are snapshots of model state used to resume after a failure and to evaluate progress. Checkpoints are large and bursty: a big model can write an enormous amount of state in a short window, and while it does, the GPUs may sit idle waiting for the write to complete. Storage that cannot absorb these write bursts turns checkpointing into a tax on utilization. The usual response is to write checkpoints to a fast tier quickly, then let them drain to the durable object tier in the background, so the training loop resumes as soon as possible. Designing this write path is as important as designing the read path. Multi-tenancy and cost in a GPU cloud A neocloud is still a service provider, so its storage carries the same tenancy and economics concerns as any shared platform, with an added performance dimension. Multiple customers share the cluster, so their data and their throughput both need isolation, which is a harder problem than data isolation alone. The design space specific to shared AI infrastructure is covered in the multi-tenant AI infrastructure design guide. On cost, the durable tier’s efficiency (erasure coding, density) governs how cheaply you can hold petabytes of training data and checkpoints, while the fast tier’s cost is justified only to the extent it raises GPU utilization. Many neoclouds also span on-premises and cloud capacity, a pattern discussed in hybrid cloud data strategy for AI workloads. For providers serving regulated or sovereign customers, the private-infrastructure view is in private AI infrastructure architecture. Checklist: storage for a neocloud Size the storage for sustained aggregate read throughput across the whole cluster, not a single-stream benchmark. Ensure parallel access can serve thousands of concurrent readers without a central bottleneck. Layer a durable object tier for datasets and checkpoints behind a fast staging tier for active reads. Evaluate a direct-to-GPU data path (such as GPUDirect Storage) to cut latency and CPU overhead. Design the checkpoint write path to absorb large bursts without stalling the training loop. Protect the durable tier with erasure coding and resilience against failures and ransomware. Isolate both data and performance per tenant on shared GPU infrastructure. Optimize durable-tier cost per terabyte, and justify fast-tier spend by the GPU utilization it buys back. Putting it together For a neocloud, storage is not a supporting act; it is the throttle on how much revenue the GPUs can generate. The winning pattern is a durable, cost-efficient object tier for datasets and checkpoints, a fast tier staging active data in front of it, a data path that gets bytes into GPU memory with minimal overhead, and a write path that absorbs checkpoint bursts, all sitting on multi-tenancy that isolates performance as well as data. Neocloud demand is projected to keep growing quickly through the rest of the decade, and the providers that treat storage as a utilization problem rather than a capacity problem are the ones that will keep their GPUs, and their margins, working. Frequently asked questions Why does storage matter so much for a GPU cloud? GPUs are expensive and process data very fast, so they are only productive when data arrives fast enough. Idle GPUs are usually caused by storage that cannot sustain the throughput a parallel training job demands. Because idle time is costly across a large cluster, storage throughput directly governs utilization and therefore revenue. What is GPUDirect Storage? GPUDirect Storage, part of NVIDIA’s Magnum IO, creates a direct memory-access path between storage and GPU memory, bypassing the CPU bounce buffer that data traditionally passes through. This reduces latency and frees CPU cycles, which helps keep data-hungry training pipelines full. Why is object storage used in AI clouds if a fast file system is faster? They serve different roles. Object storage provides the durable, petabyte-scale tier for datasets, checkpoints, and model artifacts at low cost per terabyte and with a standard interface. A faster tier, such as a parallel file system or flash cache, stages the data GPUs are actively reading. Most neoclouds use both, tiering data between them. How do checkpoints affect storage design? Checkpoints are large, bursty writes that save model state during training. If storage cannot absorb the burst, GPUs sit idle waiting for the write to finish. A common approach writes checkpoints quickly to a fast tier and drains them to the durable object tier in the background, so the training loop resumes with minimal delay. Further reading Related reading: S3 storage for AI workloads, GPU Direct Storage, multi-tenant AI infrastructure design guide, and the hub, storage for service providers.
