9 GPU bottlenecks are rarely the GPUs themselves. Modern GPUs have enormous computational capacity. The real bottleneck is data delivery—moving training data from storage to GPU memory fast enough. If storage can’t deliver sufficient speed, expensive GPUs sit idle. GPU Direct Storage (GDS) addresses this. It allows GPUs to access storage directly, bypassing CPU and system memory. Data flows directly from storage to GPU memory, reducing latency and increasing throughput. For infrastructure architects, understanding GDS is critical. This post covers how GDS works, required storage infrastructure, and planning for GDS deployments. The Data Delivery Bottleneck: Why GPU Direct Storage Matters Training large models involves moving enormous data amounts through pipelines. Datasets might be terabytes. Training loads gigabytes per second from storage. Data delivery speed determines training throughput. Traditionally, the storage-to-GPU path is indirect. Storage connects through networks to compute servers. Data flows from storage to CPU memory, then through PCI to GPU memory. Each transition introduces latency and limitations. The consequence is that training workloads are I/O-bound, not GPU-bound. GPUs could process faster than pipelines deliver. When this happens, GPUs sit idle waiting for batches. Training runs slower. You pay for GPU time without full utilization. This compounds at scale. With 100 GPUs, 20% idle means wasting 20 GPUs’ capacity. Over a month, this represents significant waste. GPU Direct Storage addresses this by optimizing the path. Data flows directly to GPU memory instead of through CPU memory. Latency decreases, throughput increases, and GPUs receive data faster. How GPU Direct Storage Works: Architectural Overview GPU Direct Storage uses multiple hardware and software layers for direct GPU-to-storage movement. At the hardware level, GPUs support direct memory access (DMA)—direct data transfer without CPU involvement. Modern storage interfaces like NVMe over Fabrics (NVMe-oF) enable GPU-initiated transfers. High-speed interconnects like InfiniBand or Ethernet provide transport. At the software level, NVIDIA’s cuFile library and similar tools provide APIs for direct GPU memory access. Instead of standard file I/O through CPU, applications call cuFile functions for direct transfers. The practical benefit: data movement decouples from CPU processing. The CPU isn’t involved. Instead, the GPU and storage system negotiate directly. This frees CPU resources and enables independent pipeline operation. Importantly, GDS is most effective when storage systems are designed for it. Not all systems expose necessary interfaces or have needed performance characteristics. Storage Infrastructure Requirements for GPU Direct Storage Building GPU clusters for GDS requires specific architecture choices. First, storage must support necessary protocols. NVMe-oF (NVMe over Fabrics) is most common—storage systems expose NVMe over networks. This enables GPU-to-storage communication using fast protocols. Second, storage must have sufficient throughput to saturate interconnects. With 100 Gbps Ethernet between GPUs and storage, storage delivering only 50 Gbps underutilizes the connection. Storage must match network and GPU capabilities. Third, storage must have low enough latency. Microsecond-range solid-state latencies are acceptable. Higher latencies may be unsuitable for GDS. Fourth, storage should support your training access patterns. Training involves sequential large reads and random shuffles. Optimize for these patterns. Compatibility and Ecosystem Considerations GDS is not universally supported. Understand your compatibility requirements before planning. GDS support is primarily available for NVIDIA GPUs. AMD GPUs and other accelerators have different approaches. Using multiple accelerator types may prevent GDS for all. Storage system support varies widely. Some enterprise systems—particularly HPC-designed ones—have good support. Others lack necessary interfaces. Explicitly evaluate compatibility before deployment. Application-level support also matters. Not all training frameworks automatically support GDS. TensorFlow and PyTorch have plugins requiring explicit configuration. Standard file I/O doesn’t automatically benefit; applications need modification to use direct APIs. This complexity means GDS is most practical for organizations with standardized workloads and infrastructure. Diverse workloads across multiple accelerators may make GDS infeasible. Performance Implications and Measurement GDS can meaningfully improve training performance, but magnitude varies. For heavily I/O-bound training—where pipelines are bottlenecks and GPUs aren’t fully utilized—GDS can increase throughput 20-50%. Improvement comes from removing CPU and memory bottlenecks. For GPU-bound training—where computation limits, not delivery—GDS may provide minimal improvement. GPUs are already utilized; faster delivery doesn’t increase speed. Additionally, GDS introduces overhead. Direct storage access requires specific APIs and proper configuration. Poorly configured systems might lose performance. Measure impact through careful benchmarking. Compare jobs with and without GDS using identical data and workloads. Measure GPU utilization, throughput, and training time. This shows whether GDS benefits your workloads. Planning Storage Deployments for GPU Direct Storage If planning large-scale GPU training infrastructure with GDS support, follow these steps: First, evaluate workloads. Are they I/O-bound or GPU-bound? Heavily GPU-bound workloads may not justify GDS complexity. I/O-bound workloads make GDS strategically important with extreme performance for AI infrastructure. Second, select storage explicitly supporting GDS. This is non-negotiable. Ask vendors about support, request documentation, and test with your workloads before deploying high-performance AI storage solutions. Third, ensure capable interconnects. GDS requires high-speed, low-latency GPU-to-storage connectivity. Standard Ethernet may be insufficient; InfiniBand might be necessary with scalable AI pipeline storage. Fourth, plan configuration effort. GDS isn’t set-and-forget. Optimal performance requires careful configuration, application support, and potentially custom work. Building High-Performance Training Infrastructure GDS is one component of high-performance training infrastructure. When properly implemented, it improves efficiency and reduces time. But it’s not appropriate for all organizations. If your organization operates large-scale GPU training with I/O bottlenecks, GDS is worth evaluating. Work with vendors to understand whether it’s practical. Benchmark carefully to understand actual impact. If benefits are significant, invest in proper implementation. If GDS isn’t practical—incompatible workloads or systems—focus on other optimizations: prefetching, caching, better organization, and smarter scheduling. The most efficient organizations understand specific bottlenecks and address them. GDS is powerful for removing I/O bottlenecks when it’s the right solution. Further Reading AI Data Pipelines: Architecture, Stages, and Best Practices Tiered Storage for AI: Scalable Performance and Cost Control AI Pipeline Scalable Storage High-Performance AI Storage High-Capacity AI Storage Getting Flash Performance for Unstructured Data
