2 As AI inference moves into production, growing context windows and persistent sessions consume more of the limited high-bandwidth memory available on GPUs. Our testing shows that object storage can provide a scalable tier for KV cache, restoring cached context in milliseconds, without meaningfully affecting token generation speed. This post examines the Scality ADI lab configuration and methodology behind the test results. We explain how each number was measured, compare the performance of different data paths, and identify where moving KV cache to object storage provides an advantage, and where it doesn’t. Do the numbers hold up outside a single request? The testing produced several headline results, including: Consistent time to first token (TTFT) from Scality ADI as the stored KV cache grew from 69 GB to 8 TB, at near DRAM access speeds A 128k-token context restored in 273 ms instead of taking 2.8 seconds to recompute 42.47 GB/s of sustained throughput that approached the measured network ceiling For teams evaluating KV cache offload, however, the numbers alone aren’t enough. A result measured from a single request says little about a production serving environment, where concurrency, queueing, fabric contention and cache-tier eviction can determine whether the architecture holds up at scale. That’s why we’re sharing the hardware and software configuration, measurement definitions, data paths compared, tail latencies and operating limits behind the results. Why KV cache becomes a capacity problem A GPU serving a model holds two major resources in high-bandwidth memory (HBM): model weights and KV cache. Model weights are fixed and a constant tax, while the KV cache grows with every token of every active session, which makes the GPU memory, and not compute speed, the thing that caps how many users one GPU can serve. For example, a 70B model at 128K tokens needs roughly 40 GB of KV cache for a single request, which is more than a 40 GB A100 has left after its weights are loaded. When the cache does not fit, there are two options: drop it and recompute the prefill when that context returns, or store it somewhere else and fetch it back. Recompute is the expensive option, because prefill is the compute-heavy part of inference and real traffic recomputes byte-identical caches constantly. A multi-turn chat re-prefills the whole conversation on every iteration, a shared system prompt gets rebuilt on every request, and if using RAG, the same retrieved document gets pasted into query after query. Paging is what made the second option possible. Once vLLM partitioned the cache into fixed-size blocks addressed through a per-request table, a block becomes a portable, self-contained unit that can leave the GPU. That turns the question into where the block should live. Here is the hierarchy of tiers: TierSpeedCapacityWhat lives thereGPU HBM~TB/sTens of GBLive blocks only and ultra-fast. This is the most expensive memory in the infrastructure.Host CPU DRAM~100s GB/s~1 TBWarm blocks, one hop away, but lost on restart.Local NVMe~GB/sTens of TBNode-local reuse that survives a restart. Belongs to one machine.Object storageNetwork-boundEffectively unboundedShared across every node, and across time. In this hierarchy, object storage is the only tier reachable from every node in the GPU fleet, which is what makes a prefix computed on node one an hour ago a cache hit on another node. The classic objection to object storage has been a concern about latency. Vanilla S3 over HTTPS indeed measures access time in milliseconds, but the issue is that a cache rehydrate only has microseconds to work with. The overhead in S3 is its control path: request signing, authorization, plus metadata lookups before data can be accessed. Scality’s native AI API was designed for that purpose, to provide a leaner, lighter access path where microsecond access is the goal. The test configuration All of the numbers we reported ran on the configuration depicted and described below. ComponentConfigurationLabCustomer lab at a large service provider. HPE GPU and storage server infrastructure.GPU server1 x HPE DL380a Gen12GPU: 8 x NVIDIA RTX6000 Pro Blackwell (no NVLink)CPU: 2 x Intel Xeon 6737PMemory: 2TB DR5Network connectivity: 4 x 100G RDMA ports (2x ConnectX-6-Dx), one per railLocal storage: 4x 3.84TB NVMe (baseline path)Storage clusterScality ADI on Six-node HPE Alletra cluster6 x HPE Alletra 4110CPU: 2 x Xeon Gold 6442YMemory: 512GB DDR5Storage: 20 x 3.84TB NVMeNetwork connectivity: 4 x 100G RDMA Note: the test was performed on only 5 of the 6 servers due to a bad NIC on server 6.NetworkGPU Server connectivity to storage cluster: 4 x 100 Gigabit Ethernet links, RDMA over Converged Ethernet (RoCE). Measured fabric ceiling 48.8 GB/s. HPE SNS3700cM32 x 100GbE, single-tierStorage connectivity: RoCE lossless end-to-end, PFC on priority 3, ECN, DCQCN, DSCP-based classification)Inference stackOpen-source inference stack, plug-in to Scality ADI via NIXL connector. vLLM Core untouched: scheduler, attention, execution engine are stock.Our additions switch on with flags.NVIDIA ModelExpress model loader Loads model weights.We added an object-storage strategy that pulls weights over NIXL.Works against standard S3 or GPU-direct engines.NVIDIA NIXL Data-movement library for inference.Scality object storage plugs in underneath through S3 and AI APIsWe applied upstream fixes to the open-source components, see details below.Storage softwareScality ADI 1.0 GAStorage ProtocolsNIXL plugin moving model weights and KV cache to and from Scality ADI via four paths:S3 API over RDMA and TCPNative AI API (balances traffic across all available links).Models servedWe tested four models across a 4B–27B parameter range, to separate storage behavior from model behavior:Gemma-3 4B, Gemma-3 12B, Gemma-3 27B, Mistral 7BContext lengths14k tokens tested with Gemma-3 27B128k tokens tested with Gemma-3 4B The inference stack was open source, with no private fork and no patched engine, so the numbers reflect what a reader can assemble from public components, plus Scality’s NIXL connector, which is available on Github. We ran tests with several models that deliberately differ in size: a 4B–27B range keeps the cache-per-token footprint small, which actually understates the benefit, since a larger model produces more cache per token and therefore more to save. What we measured Time to first token (TTFT): Measured from request submission to the first output token, which includes the model’s own prefill compute, and not just the storage fetch. This matters when comparing across labs. A faster GPU shortens the compute half and makes the storage half look proportionally larger. Median and p99: Median describes the common case and p99 describes what users complain about. Tail latency is the number that decides whether an offload tier is deployable, so we report it per path. Under concurrency, not at one request: Sessions were scaled up rather than measured in isolation. Warm versus cold: A “restore” is the warm path, where the cache exists in a tier and is fetched. A “recompute” is the cold path, with no cache anywhere and full prefill. The baseline is the cold path with prefix caching disabled. Working set scaled, not sampled: Stored session cache was grown from 69 GB to 8 TB with resume time re-measured across the range, rather than measured once at one convenient size. Comparison of recompute versus KV cache on Scality ADI object storage The KV cache on object storage is not one thing, there are several data paths a stored block can take into GPU memory, and they don’t perform the same. We therefore measured several paths to determine the differences. PathMechanismBounded byRecompute (baseline)GPU rebuilds the cache from the prompt. No storage involved.GPU computeHost DRAMCache held in the GPU server’s own memory.One server’s RAM; lost on restartS3 over RDMA and TCPS3 API over TCP and RDMANetwork fabricNative AI object API over RDMA and TCPStreamlined object storage API, with S3 control-path overhead removed.Network fabric The native AI API eliminates the S3 control-path overhead we described above and RDMA removes the cost of data copying through the host CPU incurred with TCP. Results: Four places the numbers moved The results show how Scality ADI performed across the four areas that determine whether KV cache offload is practical at scale: retrieval latency, throughput, interference with live serving and capacity growth. Time to first token: Only 64 ms behind DRAM, much faster than recomputeTime to first token from Scality ADI stays close to host DRAM across context lengths, from 14K tokens up to 128K. Figure: Time-to-First-Token by tier versus full recompute As we see from the chart, GPU HBM and host DRAM are the fastest tiers but also the smallest, so at some level of concurrency they become the limiting factor for the KV cache. This is why we say inference is a memory problem, not a compute resource problem. The key TTFT results across GPU HBM, host DRAM and Scality ADI object storage are: 14k-token context test: 166 ms restored from Scality ADI object storage, 102 ms from host DRAM, 2.3–3 s to recompute (tested with Gemma-3 27B) Also tested with a 128k-token context (Gemma-3 4B): 273 ms restored from object storage, 2.8 s to recompute Token pacing stayed flat as the cache moved off the GPU. The key comparison is restore versus DRAM access time. At 14K tokens, going to a six-node cluster across the fabric cost about 64 ms more than reading from the GPU server’s own memory, and in exchange removed the capacity ceiling that server memory imposes. That’s the core value of this architecture: unbounded scalability with near-DRAM performance for the KV cache. Throughput, and where the ceiling actually wasThe limiter in this test was the network, not the storage software. Object reads sustained 42.47 GB/s from the six-node cluster across four 100 GbE links over RoCE, this is 87% of the lab’s measured 48.8 GB/s fabric ceiling, while the storage servers stayed roughly 96% idle. At 42.47 GB/s the storage nodes still had headroom, whereas the network fabric did not. Writes under live servingReads are half of this problem, since most savvy operations people will know that large writes can impact live traffic. Our test showed that a 51 GiB model checkpoint was written to storage over RDMA while the model kept serving. Ninety-ninth-percentile (p99) token latency moved by just 1 ms, at roughly 8x less server CPU per byte than the same transfer over TCP. Capacity: 69 GB to 8 TB while response time remains flat To demonstrate performance as KV cache capacity requirements increase, the stored session cache was scaled from 69 GB to 8 TB. With this larger cache size the time to resume a conversation stayed flat across the entire range. An 8 TB KV cache represents 83X this particular lab’s GPU memory size. The key result is that the resume time did not degrade as the working set grew by two orders of magnitude, and this is the property that determines whether a cache tier will keep working as it fills up. Figure: TTFT from Scality ADI remains flat as the KV Cache grows Where KV cache offload does not payOffload is not free, and it’s not universally the right call. Its value depends on whether retrieving cached context consumes less time and compute than rebuilding it, as well as how often that cached data can be reused. It may provide little advantage in these situations: Short, unique contexts: Below a certain context-length crossover point, fetching the cache can take longer than rebuilding it, making recompute the faster option. Low reuse: If prefixes are rarely seen more than once, that’s paying to store cache nobody reads. Reuse frequency, not context length, determines how often the savings are realized. Single-node deployments: Much of the advantage here is fleet-wide sharing. On a single machine, local NVMe alone can be adequate. What we changed upstream Three public contributions came out of this work: A KV cache over-fetch fix in vLLM cut data moved per request from 5.1x to 1.1x of what was actually needed, dropping warm time to first token from roughly 370 ms to 166 ms for any storage behind NIXL, not only Scality. Object-storage weight loading was added to NVIDIA’s ModelExpress. Scality’s NIXL connector was published at github.com/scality/nixl. We view keeping these contributions in the open-source stack as critical. An accelerated data path that lives in a vendor’s private fork ties upgrades to that vendor’s release schedule. One that lives in the open stack moves on the community’s timeline. What this changes for teams running inference today GPU HBM and host DRAM continue to keep the hottest KV-cache data; that doesn’t change. What changes is the capacity ceiling behind them. RDMA-capable object storage now offers a shared, persistent, effectively unbounded tier that restores cached context far faster than recompute, without meaningfully slowing token generation as the working set grows. Plan capacity around the network, not the storage layer. In this testing, the fabric hit its limit first. What the results mean for AI inference infrastructure The role of object storage in AI infrastructure is expanding. Beyond storing training data, models and checkpoints, it can become part of the live inference memory hierarchy for large contexts, reusable prefixes and persistent sessions distributed across a GPU fleet. The right architecture will depend on model size, context length, cache reuse, concurrency and network design. See the related press release for our headline findings and broader announcement:Scality turns object storage into scalable memory for AI inference. Explore the open-source Scality NIXL connector on GitHub.
