1 Splunk SmartStore storage changes how a Splunk deployment uses disk. In a classic deployment, every indexer stores its own hot, warm and cold data on local or attached storage, and indexer clusters replicate that data between peers. With SmartStore, warm data lives in a remote object store that speaks the S3 API, and each indexer keeps only a local cache of the data most likely to be searched. Compute and storage scale separately, retention can grow without adding indexers and the object store becomes the durable home for most of the data. This article explains what Splunk SmartStore storage requires: the remote object store, the local cache, the network between them and how to think about durability, performance and cost. It is written for SIEM and SOC platform engineers and security architects, and draws on Splunk’s published SmartStore documentation. How SmartStore works In a SmartStore-enabled index: Hot buckets are written locally on each indexer as data arrives and are replicated within the indexer cluster as usual. When a bucket rolls from hot to warm, the indexer uploads it to the remote store. The remote copy becomes the master copy. The cache manager on each indexer keeps recently used warm buckets locally and evicts older ones as space is needed. When a search needs a bucket that is not in cache, the indexer fetches it from the remote store. Because the remote store holds the durable copy of warm data, the indexer cluster no longer needs to keep multiple replicated copies of warm buckets on local disks. That reduces local storage requirements and makes it easier to replace or add indexers. Remote store requirements Splunk documents support for Amazon S3, including S3 API-compliant on-premises object stores, as well as Google Cloud Storage and Azure Blob Storage. For on-premises deployments, the remote store is typically an S3-compatible object storage platform in the organization’s own data center. Key requirements for the remote store include: S3 API compatibility for the operations SmartStore uses, validated against Splunk’s requirements and ideally through vendor testing with Splunk. Durability. The remote store holds the master copy of warm data, so it must protect against drive, node and ideally site failures. See data durability in high-density storage systems. Throughput. Uploads happen continuously as buckets roll, and cache misses during searches can trigger many parallel downloads. The store must sustain both. Scale. Security data grows quickly, and longer retention multiplies it. The store should scale by adding nodes. Security. Encryption, access control and audit logging for sensitive security data. Object storage built for scale-out workloads fits these requirements. See object storage for your data lake for related design principles. Local cache requirements The cache is where search performance is won or lost. Splunk’s documentation recommends provisioning local storage on each indexer for the equivalent of about 30 days of indexed data, with a minimum of seven to ten days. For Splunk Enterprise Security, the documentation recommends about 90 days. As an example, Splunk notes that an indexer adding about 100 GB per day should reserve about 3 TB for cached data. Splunk recommends SSD storage for the cache on Linux servers, and specific SSD-backed instance types in public clouds. The cache must hold hot buckets as well as cached warm buckets, so size it with both in mind. Our article on how to size the SmartStore cache vs remote storage works through the calculation. Network requirements Splunk’s documentation recommends 10 Gbps network connectivity from each indexer to the remote store for optimal performance. In practice, the network must handle: Continuous uploads of rolled buckets from every indexer. Bursts of downloads when searches cover older data that is not cached. Rebalancing or bootstrapping when indexers are added or replaced. On-premises, that usually means placing the object store on the same high-speed network as the indexers. In the cloud, keep indexers and the remote store in the same region to avoid latency and data transfer charges. Why organizations adopt SmartStore Decoupled compute and storage Retention requirements for security data keep growing. In a classic deployment, keeping more data means adding indexers, even if search load has not changed. SmartStore lets you add object storage capacity for retention and indexers only for ingest and search load. Lower local storage needs Without replicated warm copies on every indexer, local storage requirements fall. Indexers become easier to replace, and hardware can be standardized on fast cache storage. Faster recovery and scaling Because the durable copy lives in the remote store, a failed indexer can be replaced and repopulate its cache from the remote store, rather than requiring a full data rebuild from peers. Longer retention at lower cost Object storage generally costs less per terabyte than the high-performance storage used on indexers, which makes longer retention more affordable. That matters as regulators and incident responders ask for more history. See how long security logs should be retained. On-premises or cloud remote store The remote store can be in a public cloud or on premises. The decision depends on where indexers run, cost over time, data residency and how often older data is searched. Searches that pull large volumes from a cloud object store can incur request and data transfer costs if indexers are outside the cloud region, while on-premises object storage has predictable costs but requires capacity planning. Our comparison of SmartStore on-prem vs AWS S3 costs covers the trade-offs. Durability and protection The remote store becomes critical infrastructure for the SOC. Losing it would mean losing historical security data needed for investigations and compliance. Good practice includes: Erasure coding across nodes so the store survives drive and node failures without data loss. See erasure coding vs replication. Multi-site protection through replication or a platform that spans sites. Immutability where appropriate, so attackers cannot delete security evidence. Attackers who gain access often try to remove logs that would reveal their activity. See S3 object lock: immutability and WORM. Separate credentials for the remote store, with access limited to the indexers and administrators who need it. Performance expectations With SmartStore, searches over recent data run from local cache and perform as they would on a classic deployment. Searches over older data that is not cached depend on how quickly buckets can be fetched from the remote store, which depends on network bandwidth, remote store throughput and bucket sizes. Organizations that frequently run long-range searches, such as threat hunts over months of data, should size both the cache and the remote store for that pattern. The scality.com blog’s post on S3 request performance covers the storage factors involved. Operating the remote store Day-to-day operation of the remote store is mostly about watching capacity, throughput and errors. Track upload backlogs on indexers, which signal that the remote store or network cannot keep up with bucket rolls, and track cache miss rates, which show whether searches are pulling more data from remote storage than expected. Plan remote store expansion well ahead of need, since security ingest can jump when new data sources are onboarded. Migrating to SmartStore Existing deployments can migrate indexes to SmartStore, uploading existing warm and cold buckets to the remote store. Plan for the upload volume, network bandwidth and time required, and test with a non-critical index first. Follow Splunk’s migration documentation for the version you run, since procedures and constraints can change between releases. Checklist: Splunk SmartStore storage Choose a remote store type supported by Splunk and validate S3 compatibility. Ensure the remote store is durable across drives, nodes and ideally sites. Size local SSD cache for about 30 days of data, or 90 days for Enterprise Security. Provide 10 Gbps connectivity from each indexer to the remote store. Plan remote store capacity for retention growth. Protect the remote store with separate credentials and, where appropriate, immutability. Test long-range search performance with cache misses. Model cost for on-premises and cloud options. Plan and test migration of existing indexes. Putting it together Splunk SmartStore storage has two halves: a fast local cache on each indexer and a durable, scalable S3-compatible remote store for warm data. Size the cache for the time range most searches cover, connect indexers to the remote store with high bandwidth and choose a remote store that can grow with retention while protecting security data against failure and tampering. Done well, SmartStore lets you keep more security history for less, scale indexers only when search load demands it and recover from indexer failures faster. Frequently asked questions What is Splunk SmartStore? SmartStore is an indexer feature that stores warm data in a remote object store and keeps a local cache of recently used data on each indexer. Can SmartStore use on-premises object storage? Yes. Splunk documents support for S3 API-compliant on-premises object stores as well as public cloud options. How much cache does SmartStore need? Splunk recommends local storage for about 30 days of indexed data per indexer, at least seven to ten days, and about 90 days for Enterprise Security. Does SmartStore still replicate data between indexers? Hot buckets are still replicated within the cluster. Warm buckets rely on the remote store for durability, so multiple local replicas are not needed. What network bandwidth does SmartStore need? Splunk recommends 10 Gbps connectivity from each indexer to the remote store for optimal performance. Further reading See SmartStore cache sizing, security log retention, security data lakes, SmartStore on-prem vs AWS S3 costs and what is SIEM.
