4 Open table formats on on-prem S3 storage are now a common foundation for data lakehouses that stay inside the organization’s own data centers. An open table format adds database-like features to collections of files in object storage: schemas, transactions, snapshots and efficient query planning. The leading formats are open source and designed to work with any storage that exposes the S3 API, which means they are not tied to a public cloud. The short answer to whether open table formats work on S3-compatible storage on premises is yes, provided the storage meets a few requirements. This article explains how table formats use object storage, what the storage must support, how catalogs fit in, which maintenance tasks matter and how to test a deployment. For the wider architecture, see our hub on running a data lakehouse on on-prem object storage. What an open table format does Raw files in object storage, such as Parquet or ORC, have no notion of a table. Early data lakes relied on directory layouts and file listings to define tables, which made changes risky and queries slow at scale. Open table formats solve this by adding a metadata layer: Table metadata records the schema, partitioning and current snapshot. Manifests list the data files in each snapshot, with statistics such as row counts and column minimum and maximum values. Snapshots capture the state of the table at each commit, enabling time travel and rollback. Atomic commits ensure readers see either the old table or the new one, never a partial write. Query engines use this metadata to plan queries without listing directories and to skip files that cannot match a filter. See ACID transactions for the underlying concepts. How table formats use object storage All data and metadata live as objects. A typical write: Writes new data files to object storage. Writes new manifest files describing them. Writes a new table metadata file pointing to the new snapshot. Atomically updates the catalog so the table points to the new metadata file. Reads follow the pointer from the catalog to the metadata file, then to manifests, then to the data files they need. Because the catalog swap is the commit point, the object store itself does not need to support transactions. It needs to store and serve objects reliably and consistently. What on-prem S3 storage must support Strong read-after-write consistency When a writer commits a snapshot, readers must be able to see the new metadata and data files immediately. Storage with eventual consistency can cause readers to miss files or fail queries. Confirm that the storage provides strong consistency for new objects, overwrites and listings. Core S3 API coverage Table formats and engines rely on standard operations: PUT, GET with byte ranges, HEAD, DELETE, multi-object delete, multipart upload and listing. Ranged GETs are especially important because columnar readers fetch specific column chunks. Test error handling as well, since clients retry based on specific error codes. See S3 error codes. Performance for small and large objects Metadata and manifest files are small and read often; data files are larger and read in parallel ranged requests. Storage should handle high request rates with low latency as well as high aggregate throughput. See object storage performance for SQL query engines. Scale in object count Large tables accumulate millions of data files and many metadata files over time. Storage must handle large object counts per bucket without slowing listings or lookups. Security and access control Buckets for analytics often hold sensitive data. Look for fine-grained bucket policies, identity integration, encryption at rest and in transit and audit logging. See S3 access policies. The role of the catalog Every open table format needs a catalog to track the current metadata location for each table and to perform the atomic swap at commit. Options include REST-based catalog services, metastore services carried over from older data lake platforms and database-backed catalogs. On premises, the catalog runs as a service alongside the query engines, typically backed by a database that must itself be highly available and backed up. The catalog is small, but it is critical: if it is lost, tables cannot be found even though all data is intact in object storage. Maintenance tasks that affect storage Open table formats generate files over time. Regular maintenance keeps performance and capacity in check: Compaction merges small data files into larger ones, reducing request counts and improving scan speed. Snapshot expiration removes old snapshots beyond the retention you need for time travel and audit. Orphan file cleanup deletes files no longer referenced by any snapshot, such as leftovers from failed writes. Manifest rewriting consolidates metadata for faster planning. Retention for time travel should be set deliberately. Keeping every snapshot forever consumes capacity; expiring too aggressively removes the ability to audit or roll back. Bucket versioning on the object store adds another layer of protection but also increases capacity use. See S3 versioning costs. Common issues on premises and how to avoid them Path-style vs virtual-hosted-style addressing: configure clients to match the storage endpoint, often using path-style access with on-premises endpoints. TLS and certificates: internal certificate authorities must be trusted by every engine and catalog client. Endpoint and region settings: clients often expect a region value even on premises; set it consistently. Timeouts and retries: tune client connection pools and retry settings for high-concurrency workloads. Clock skew: request signing fails if clocks drift, so keep time synchronized across compute and storage. Choosing a table format The main open table formats share the same core ideas but differ in ecosystem support, how they handle updates and deletes and how mature their tooling is for specific engines. From the storage side the requirements are almost identical, so the choice usually comes down to which engines your teams use, how often data is updated in place and which catalog you plan to run. Many organizations standardize on one format for new tables while keeping the option to read others. Because the data sits in open file formats on S3-compatible storage either way, changing format later is a data conversion project rather than a storage migration. Moving from older table layouts Many organizations already have data in object storage or in a legacy data lake using directory-based tables. These can usually be converted in place or migrated incrementally: In-place registration adds table format metadata over existing Parquet or ORC files without rewriting them, when the file layout allows. Rewrite on migration copies data into new tables with better file sizes, partitioning and sorting, which often improves performance. Dual running keeps old and new tables in parallel while queries and pipelines are switched over and results are compared. Plan the catalog first, so new tables are registered in the system that engines will use going forward. Testing a deployment Before moving production tables, run a structured test: Create tables with realistic schemas, partitioning and file sizes. Run concurrent writes and reads to confirm consistency and commit behavior. Execute representative queries and measure planning time and scan throughput. Run maintenance jobs and confirm that deletes, compaction and snapshot expiration work as expected. Simulate failures, such as losing a storage node or restarting the catalog, and confirm recovery. Sovereignty and control Because open table formats are open and storage-agnostic, they let organizations keep analytics data on infrastructure they control without giving up modern lakehouse features. That matters for regulated industries and public sector bodies with data residency requirements. It also reduces lock-in, since the same tables can be read by multiple engines. See sovereign data lakes and migrating from HDFS to object storage. Checklist: open table formats on on-prem S3 Strong read-after-write consistency, including listings. Full coverage of core S3 operations, including ranged GETs and multipart uploads. High request rates and low latency for small metadata objects. High aggregate throughput for parallel data reads. Scale to millions of objects per bucket. A highly available, backed-up catalog service. Scheduled compaction, snapshot expiration and orphan file cleanup. Correct client settings for addressing style, TLS, region and timeouts. Fine-grained access control, encryption and audit logging. Putting it together Open table formats work well on S3-compatible object storage on premises. The storage must be consistent, compatible with the S3 operations table formats use and fast for both small metadata reads and large parallel scans. Add a reliable catalog and regular maintenance and the result is a lakehouse with modern features that stays under your control. For the full picture, see running a data lakehouse on on-prem object storage. Frequently asked questions Do open table formats require public cloud object storage? No. They work with any storage that implements the S3 API with strong consistency, including on-premises object storage. What S3 features do open table formats need? Strong consistency, standard object operations, ranged GETs, multipart uploads, multi-object delete and efficient listing. Does the object store need to support transactions? No. Commits are made atomic by the catalog, which swaps a pointer to new metadata. The object store must store and serve objects reliably and consistently. What maintenance do open table formats need? Compaction, snapshot expiration, orphan file cleanup and periodic manifest rewriting. Is a catalog required? Yes. The catalog tracks the current metadata for each table and performs the atomic commit.
