Amazon S3 Tables vs Self-Managed Iceberg

Apache Iceberg has become the default open table format for data lakehouses. Databricks, Snowflake, AWS, and Google Cloud all treat it as the interoperability standard. If you are running analytics on AWS in 2026, Iceberg is likely already in your stack or on your roadmap.

But choosing the format is only half the decision. On AWS, you now face a second question: how do you operate your Iceberg tables in production? There are three approaches, and most teams end up evaluating all of them:

1. 1.Amazon S3 Tables — AWS embeds Iceberg directly into S3 with automatic compaction, snapshot expiration, and orphan cleanup. Zero maintenance infrastructure, but limited visibility and engine support.
2. 2.Self-managed Iceberg — you run standard S3 + Glue (or a REST catalog) and own every piece of the operational stack: compaction, cleanup, monitoring, scheduling, and failure recovery.
3. 3.Control plane (LakeOps) — an autonomous orchestration layer that connects to any catalog (including S3 Tables and Glue), runs the full maintenance sequence with a purpose-built Rust engine, and provides health-classified observability, policy governance, multi-engine routing, and query-aware optimization across the entire estate.

Each approach makes a different trade-off between automation, visibility, control, and operational maturity. This guide compares all three across every dimension that matters in production — with a decision framework at the end.

What is Amazon S3 Tables?

Amazon S3 Tables, announced at re:Invent 2024 and now generally available across most AWS regions, is a new S3 bucket type designed specifically for Apache Iceberg tables. Instead of storing arbitrary objects, an S3 table bucket stores structured Iceberg tables with a built-in REST Catalog endpoint, automatic maintenance, and native integration with AWS analytics services.

When you create a table bucket, AWS handles the Iceberg catalog internally. Tables are created through the S3 Tables API, Athena, or EMR. The service manages metadata pointers, manifest files, and atomic commits behind the scenes. Each table bucket supports up to 10,000 tables.

The key value proposition: S3 Tables runs automatic compaction, snapshot expiration, and orphan file cleanup as background processes with zero configuration. AWS reports up to 10x higher transactions per second compared to general-purpose S3 buckets and up to 3x faster query performance through continuous optimization.

S3 Tables also supports Intelligent-Tiering for automatic storage cost optimization and cross-region replication for disaster recovery and global analytics workloads.

What is self-managed Iceberg?

Self-managed Iceberg is the traditional approach: you write Iceberg tables to standard S3 buckets, register them in AWS Glue Data Catalog (or a REST catalog like Polaris, Nessie, or Gravitino), and run maintenance operations yourself using Spark, Athena, or Glue ETL.

You own every piece of the stack. Compaction is a scheduled Spark job or Athena OPTIMIZE command. Snapshot expiration is a stored procedure call. Orphan file cleanup is a separate pipeline. Catalog management, IAM policies, Lake Formation rules, engine configuration — all yours to configure, monitor, and debug.

This is how the vast majority of production Iceberg deployments run today. It works with every Iceberg-compatible engine (Trino, Spark, Flink, DuckDB, Snowflake, Databricks, dbt) on day one, because it follows the open Iceberg specification without proprietary endpoints.

What is LakeOps?

LakeOps is an autonomous control plane for Apache Iceberg. It sits above the storage layer — connecting to any catalog (Glue, REST/Polaris, Nessie, S3 Tables, Gravitino) — and orchestrates everything that neither S3 Tables nor self-managed Iceberg handles: the full maintenance lifecycle in correct dependency order, query-aware compaction on a purpose-built Rust/DataFusion engine, health-classified observability across every table and catalog, cascade policy governance, multi-engine query routing, and AI agent access via MCP. It works with both S3 Tables and standard S3 simultaneously.

Architecture comparison

S3 Tables merges the storage layer and the Iceberg catalog into a single managed service. Self-managed Iceberg keeps them separate — S3 for storage, Glue (or another catalog) for metadata, and your own compute for maintenance. LakeOps adds the operational control plane above either storage approach.

How each approach compares across every operational dimension:

Compaction and maintenance

Maintenance is the biggest operational difference between the two approaches — and the most common reason teams consider S3 Tables in the first place.

S3 Tables: managed maintenance

S3 Tables runs compaction, snapshot expiration, and orphan file cleanup as automatic background processes. You configure the target file size (64 MB to 512 MB, default 512 MB) and compaction strategy (binpack, sort, or z-order) via the PutTableMaintenanceConfiguration API. Everything else — scheduling, triggering, execution — is handled by AWS.

The upside: zero maintenance infrastructure. No Spark clusters for compaction, no Airflow DAGs for scheduling, no on-call for failed maintenance jobs. For teams that want storage to be infrastructure rather than an operational concern, this is a genuine advantage.

The downside: it is a black box. AWS does not expose detailed compaction logs or let you tune the schedule beyond basic table-level settings. You cannot control when compaction runs, inspect what it did, or correlate maintenance events with query performance changes. The trigger conditions for auto-compaction are not documented — understanding the internal logic requires opening a support case.

For teams with streaming freshness SLAs, this opacity is a problem. Compaction that runs at the wrong time can temporarily increase query latency. Without visibility into the schedule, you cannot coordinate maintenance windows with business-critical query patterns.

Self-managed: full control, full burden

With self-managed Iceberg, you own the entire maintenance lifecycle. Compaction is typically a Spark stored procedure (rewrite_data_files), an Athena `OPTIMIZE` command, or a Glue table optimizer. Snapshot expiration and orphan cleanup are separate procedure calls. You control the schedule, the strategy, the target file size, and the execution environment.

You also own the failures. A failed compaction job at 3 AM does not fix itself. A snapshot expiration that runs before orphan cleanup risks referencing files that no longer exist. The maintenance sequence matters — expire snapshots first, then clean orphans, then compact, then rewrite manifests, then compute statistics — and getting it wrong wastes compute or risks data loss.

At small scale (10–50 tables), this is manageable with a few Airflow DAGs. At production scale (hundreds or thousands of tables with different ingestion rates, query patterns, and SLAs), manual maintenance becomes the primary source of operational incidents.

Control plane: autonomous maintenance with full visibility

S3 Tables automates execution but sacrifices visibility. DIY maintenance provides control but does not scale. LakeOps combines both: automated execution with full observability, policy-based governance, and cross-engine awareness.

Building this orchestration layer yourself is a significant engineering undertaking. You need at least: a health scorer that classifies every table by file count, manifest depth, snapshot accumulation, and partition skew. A sequencer that runs the correct maintenance order — expire snapshots → remove orphan files → compact data files → rewrite manifests → compute statistics — because misordering risks data loss or wasted compute. A compaction engine fast enough to process petabytes without blocking queries. A query analyzer that tracks field access patterns to inform sort order. A policy engine with cascade logic from table to namespace to catalog. An observability pipeline that correlates maintenance events with query performance. A multi-catalog connector for Glue, REST, S3 Tables, and HMS simultaneously. An engine router. And an audit system. Most teams never finish building the first three.

Rust-powered, query-aware compaction. The execution engine is built in Rust on Apache DataFusion, processing Parquet files directly without JVM overhead, shuffle stages, or cluster spin-up. Binpack compaction completes in 221 seconds versus 1,612 seconds for Spark on 200 GB tables — roughly 7x faster at a fraction of the cost ($5/TB vs $50/TB). Sort compaction goes further: LakeOps analyzes actual query patterns across engines to determine the optimal sort order per table, runs layout simulations that predict scan reduction before writing a single byte, and applies Z-order or sort strategies that match how the table is actually queried — not how an engineer guessed it would be queried six months ago.

Health-signal-triggered, not cron-based. Fixed schedules either run too often (wasting compute) or too infrequently (letting tables degrade). LakeOps classifies every table into Critical, Warning, or Healthy tiers based on file count, average file size, delete file ratio, manifest depth, and snapshot accumulation. Maintenance triggers when health signals cross policy thresholds, so each table gets the operations it actually needs.

Full maintenance sequence, fully auditable. Every operation — expire snapshots, remove orphan files, compact data files, rewrite manifests, compute column-level statistics — runs in the correct dependency order with full before/after metrics: file counts, sizes, durations, and status. Unlike S3 Tables, every action is inspectable. Unlike manual scripts, the sequence is guaranteed correct.

Policy cascade replaces per-table DAGs. Governance policies are defined at three levels — table overrides, namespace defaults, catalog baseline — and cascade automatically. One compaction policy on a catalog applies to every table in every namespace unless overridden. This replaces the hundreds of individual Airflow DAGs that teams accumulate as their lake grows, and ensures new tables are governed from the moment they appear.

Observability and table health

Observability is where the three approaches diverge most sharply in day-to-day operations.

S3 Tables provides basic monitoring through CloudWatch metrics — table size, object count, and maintenance job status. No cross-table health dashboard, no severity-ranked insights, no correlation between maintenance events and query performance, and no visibility into why compaction ran (or did not run) for a specific table.

Self-managed (DIY) gives you whatever observability you build. Most teams start with nothing and add monitoring reactively after the first incident — a query that suddenly takes 10x longer because a table accumulated 50,000 small files, or an S3 bill that doubled because orphan files were never cleaned up.

LakeOps provides production-grade observability out of the box: table-level health classification (Critical, Warning, Healthy) based on file count, manifest depth, snapshot accumulation, orphan volume, and partition skew. Severity-ranked Insights surface degradation before it reaches invoices or query dashboards. Every maintenance operation is fully auditable with before/after metrics.

Tables in S3 table buckets and tables in Glue appear in the same health dashboard, with the same severity classification and the same maintenance audit trail. For organizations running mixed estates — some tables in S3 Tables, others in Glue — this unified view is the only way to get consistent operational visibility.

Engine compatibility and tooling

Engine support is the dimension where self-managed Iceberg has a clear structural advantage today.

Self-managed Iceberg works with every Iceberg-compatible engine on day one: Trino, Spark, Flink, DuckDB, Snowflake, Databricks, ClickHouse, StarRocks, and dbt. New tools integrate against the open Iceberg spec first — when a query engine ships Iceberg support, it works against a standard Glue-registered table immediately.

S3 Tables requires engines to support the proprietary s3tablescatalog endpoint. Athena, EMR, and Redshift work natively. Spark and Trino work through the REST Catalog API. But ecosystem gaps remain:

• dbt: As of August 2026, dbt support for S3 Tables remains experimental. The dbt-athena adapter requires special materialization logic because S3 Tables does not support ALTER TABLE RENAME — the standard near-zero-downtime swap used for Glue Iceberg tables. Tables must be dropped and recreated instead.
• Athena views: CREATE VIEW is not supported against the s3tablescatalog endpoint. Views are fundamental to most transformation and BI layers.
• Catalog separation: S3 table bucket catalogs and existing Glue catalogs cannot be unified into a single engine catalog. Cross-catalog queries are required for joins between tables in different catalog types.
• DDL limitations: Some DDL operations are restricted on REST-type catalogs, requiring a Spark session for table creation in scenarios where Trino or Athena would normally suffice.

For teams where dbt is the primary transformation framework — which is most modern data teams — this gap alone may be decisive.

Security and governance

S3 Tables uses table-level IAM resource policies for access control. Straightforward for small teams, but lacks the granularity that enterprise deployments require.

Self-managed (DIY) integrates with AWS Lake Formation, which provides column-level security, row-level filtering, Tag-Based Access Control (LF-TBAC), and cross-engine enforcement across Athena, EMR, and Redshift. More complex to configure (especially via Terraform), but it provides the enterprise-grade access control that regulated industries require. If your compliance requirements include column-level masking or row-level filters, self-managed Iceberg with Lake Formation is the only data-access-control option on AWS today.

LakeOps works with Lake Formation for data access control and adds the layer that both other approaches lack: operational governance. LakeOps Policies define compaction thresholds, retention windows, and cleanup schedules per table, namespace, or catalog — with cascade logic, version history, and audit trails. One compaction policy on a catalog applies to every table unless overridden at a more specific scope. This is the governance layer for maintenance operations, complementing Lake Formation's governance layer for data access.

Cost comparison

Cost is often the first question teams ask, but the answer depends on workload characteristics.

S3 Tables pricing

S3 Tables storage costs $0.0265 per GB/month (15% higher than standard S3 at $0.023). Maintenance operations are priced at:

• Compaction processing: $0.005 per GB processed (binpack), higher for sort/z-order — reduced 90% in July 2025
• Object processing: $0.002 per 1,000 objects
• Object monitoring: $0.025 per 1,000 objects/month (continuous, scales with file count)

The monitoring fee is the hidden cost driver. It is small per object but never goes away and scales linearly with your file count. For tables with millions of small files (common in streaming workloads before compaction catches up), this adds up.

Self-managed pricing

Standard S3 at $0.023 per GB/month, plus the compute cost of running maintenance:

• Spark compaction on EMR: varies by instance type and cluster size, but typically $0.10–0.50 per TB processed
• Athena OPTIMIZE: $5 per TB scanned
• Glue table optimizers: included in Glue pricing

The hidden cost of self-managed is not the compute — it is the engineering time. Building, monitoring, and maintaining the maintenance infrastructure across hundreds of tables requires ongoing engineering investment that does not show up on the AWS bill but shows up on the team's capacity.

LakeOps pricing

LakeOps uses standard S3 storage pricing (no 15% surcharge). The Rust/DataFusion compaction engine processes Parquet directly without JVM overhead, at roughly $5/TB versus $50/TB for Spark-based maintenance. On a 1 TB TPC-DS benchmark: S3 Tables took 6,300 seconds, Spark took 1,612 seconds, and the LakeOps Rust engine completed binpack in 221 seconds — roughly 29x faster than S3 Tables and 7x faster than Spark. That difference compounds linearly with estate size.

The full picture

For small greenfield deployments (under 50 tables, moderate write rates), S3 Tables is the simplest starting point. For teams that already have Spark infrastructure and want full control, self-managed works — until the maintenance burden outpaces the team. For anything at production scale, the control plane approach is the most cost-effective: faster compaction at a fraction of the cost, zero engineering time on maintenance infrastructure, and the operational visibility to catch cost waste before it reaches the invoice.

Multi-engine query routing

Production data estates rarely use a single query engine. Athena handles ad-hoc SQL. Spark runs ETL pipelines. Trino powers dashboards. DuckDB handles exploratory analysis. Each engine has different cost characteristics, latency profiles, and optimal workload types.

Neither S3 Tables nor self-managed Iceberg includes a query routing layer. Teams typically hardcode engine selection in application code or maintain manual runbooks. A proper routing layer requires SQL dialect translation, health-aware failover, cost-based engine selection, and stable endpoint management — a non-trivial system on its own.

LakeOps provides a unified routing layer that dispatches queries to the best engine based on cost, latency, and table health. Routing groups map workload types (analytics, BI, ETL, AI agents) to engine pools with priority and failover. Applications get a stable endpoint URL — when you add a faster engine or retire an old one, the routing layer absorbs the change without application code updates.

This is also where AI agent access enters the picture. LakeOps exposes Iceberg tables through a Model Context Protocol (MCP) interface — AI agents query the lake through the same routing and governance layer as human analysts, with guardrails, cost limits, and row-level controls enforced automatically.

Decision framework: which architecture fits your situation?

The right choice depends on where you are today, not where you want to be eventually:

Choose S3 Tables when: - You are building a **greenfield** Iceberg deployment on AWS with no existing Glue catalog - Your primary engines are **Athena, EMR, and Redshift** — no dbt, no Trino-primary stack - Your team has **no appetite for maintenance infrastructure** and can tolerate black-box compaction - Your workload is **high-throughput streaming** with heavy concurrent writes and you need the higher TPS - You are **AWS-only** with no multi-cloud requirements - Your security model is **table-level IAM** — no column-level or row-level access control needed

Choose self-managed Iceberg when: - You have an **existing Glue catalog** with tables already in production - **dbt** is your primary transformation framework - You need **Lake Formation** for column-level, row-level, or tag-based access control - You run a **multi-engine** stack (Trino, Spark, Flink, DuckDB, Snowflake) and need day-one compatibility - You need **multi-cloud portability** — data must be accessible from Azure, GCP, or on-premises - You require **full maintenance visibility** — knowing exactly when compaction ran, what it did, and how it impacted query performance - You operate at **scale** (hundreds or thousands of tables) where per-table tuning and policy governance matter

Add LakeOps when: - You want **autonomous maintenance** without building the infrastructure yourself — regardless of whether tables are in S3 Tables, Glue, or both - You need **observability** across the estate — health classification, maintenance audit trails, and degradation alerts - You run **multiple engines** and need cost-aware routing with stable endpoints - You need **query-aware compaction** — sort orders based on actual access patterns, not guesses - You need **operational governance** — cascade policies from catalog to namespace to table - You are scaling past the point where Airflow DAGs and manual maintenance are sustainable

In practice, most production teams combine a storage choice (S3 Tables, standard S3, or both) with LakeOps as the operational layer. The storage decision determines where data lives; LakeOps determines how it is maintained, monitored, and accessed.

How the three approaches work together

S3 Tables, self-managed Iceberg, and LakeOps are not competing alternatives — they operate at different layers of the stack, and the most effective deployments combine them.

S3 Tables handles storage-layer automation for tables that benefit from managed compaction and high write TPS. Self-managed Iceberg on standard S3 handles tables that need full engine compatibility, Lake Formation security, or multi-cloud portability. LakeOps handles the operational layer above both — the observability, maintenance orchestration, policy governance, query routing, and AI enablement that neither storage approach provides.

The point is not any single capability — it is having them work together as a coherent system. Maintenance decisions are informed by health data. Compaction strategies reflect actual query patterns. Policies propagate automatically as the lake grows. Tables in S3 Tables and tables in Glue appear in the same dashboard with the same governance. For self-managed tables, LakeOps replaces the manual maintenance infrastructure with autonomous, policy-driven optimization that runs the correct sequence at native speed.

Operationally, the onboarding is straightforward: connect your Iceberg catalogs (Glue, S3 Tables, REST), review the health assessment LakeOps generates on connection, set policies at whatever scope makes sense, and switch to autonomous mode. No Spark clusters to provision, no DAGs to write, no maintenance runbooks to maintain.

Watch demo·LakeOps product demoYouTube ↗

Summary

S3 Tables lowers the barrier to entry for Iceberg on AWS. Automatic compaction and managed storage are genuine advantages for greenfield deployments with straightforward requirements. The trade-off: limited engine compatibility, black-box maintenance, table-level-only security, and no cross-estate visibility.

Self-managed Iceberg remains the standard for deployments that need full engine compatibility, Lake Formation security, and multi-cloud portability. The trade-off: owning every piece of the operational stack across hundreds or thousands of tables.

LakeOps provides the operational control plane that makes either storage approach work at production scale — autonomous maintenance with a 29x-faster Rust engine, health-classified observability, cascade policy governance, query-aware optimization, multi-engine routing, and AI agent access. It works with S3 Tables, standard S3, or both simultaneously.

The question is not which one to pick — it is which combination of storage, catalog, and operational layers gives you the right balance of automation, control, and visibility. Start with the decision framework above, and connect your catalogs to LakeOps to see the full operational picture across your entire lake.

Further reading

• Managed Iceberg in 2026 — the nine components of a complete Iceberg control plane
• Iceberg Lake Compaction — deep dive into the Rust compaction engine benchmarks referenced in this article
• Autonomous Iceberg Table Maintenance — how the full maintenance sequence runs autonomously at lake scale
• Iceberg Metadata Lifecycle Maintenance — the correct maintenance sequence and why ordering matters
• Apache Iceberg on AWS S3: A Guide — the full AWS ecosystem for Iceberg: Glue, Athena, EMR, Redshift, and configuration best practices
• MCP for Iceberg with LakeOps — how AI agents access lakehouse data through the Model Context Protocol
• Iceberg vs Delta Lake — format comparison covering governance, engine support, and community trajectory
• Optimizing Iceberg for Agentic AI — preparing your lakehouse for AI agent workloads
• Iceberg Cost Optimization in 2026 — strategies for reducing storage, compute, and maintenance costs across your Iceberg estate
• Iceberg Lakehouse Observability Guide — building production-grade monitoring for table health and maintenance operations