Iceberg Compaction
Autonomous, query-driven
Iceberg compaction
LakeOps learns from your actual production queries and physically reorganizes data to match how it's read. Triggered autonomously on events, policies, or manually — and 95% faster than Spark.
Runs on your stack
Not just faster compaction —
smarter compaction
The only compaction engine that learns from your queries, runs autonomously, and delivers 95% faster performance than Spark.
Query-driven
Learns which columns your queries filter, join, and group by — then physically sorts data to match. Engines skip irrelevant files at read time.
51% less data scannedAutonomous
Triggered by events, governed by policies, or run manually. No cron jobs, no Spark clusters — compaction happens as part of automated maintenance.
Zero manual intervention95% faster
Purpose-built Rust engine with Apache DataFusion. No JVM, no GC pauses, no OOM. TB-scale tables compacted in minutes — fast enough to run continuously.
221s vs 1,612s (Spark)Pillar 1
Compaction that learns from your queries
LakeOps continuously observes which columns appear in WHERE, JOIN, and GROUP BY clauses across all engines hitting each table. During compaction, data is physically re-sorted by those columns — so Parquet row-group min/max statistics become effective and engines skip irrelevant data without reading it.
The result: drastically less I/O per query, faster response times, and lower compute costs — automatically, without anyone configuring sort keys manually.
- 51% less data scanned — sorted by real production filter columns
- 47,000 → 280 files: same data, query drops from 52s to 5.8s
- Self-improving: sort strategy evolves as query patterns change
- Per-table optimization — every table gets its own sort order
Files before
47,000
Files after
280
Query before
52s
Query after
5.8s
Scan volume reduced 51%
Sorted by actual WHERE/JOIN columns
Pillar 2
Runs autonomously — or on your terms
Compaction in LakeOps isn't a nightly batch job. It's an autonomous process that responds to your lakehouse in real time — triggered by catalog events, governed by policies you define, or kicked off manually when you need it.
- Event-driven: auto-triggers on new commits, partition updates, or threshold breaches
- Policy-based: define SLAs per table — file count, size targets, freshness windows
- Manual override: one-click compaction for any table, partition, or namespace
- Part of maintenance automation: chains with snapshot expiry, orphan cleanup, manifest rewrite
Event-driven
On new commits or threshold breach
Policy-based
SLA targets per table / namespace
Scheduled
Cron-like windows if preferred
Manual
One-click for any table or partition
Pillar 3
95% faster — built on Rust & DataFusion
No JVM. No garbage collection pauses. No OOM crashes. LakeOps compaction is a purpose-built Rust binary powered by Apache DataFusion — the same query engine behind Apache Arrow and InfluxDB 3.0. It's fast enough to run continuously, so tables never degrade between maintenance windows.
- 221s vs 1,612s (Spark) vs 6,300s (S3 Tables) on identical 200 GB dataset
- 2,522 MB/s peak throughput — TB-scale tables compacted in minutes
- Bounded memory: spills to disk gracefully, no OOM regardless of table size
- Zero-copy Arrow columnar format — no serialization between stages
200 GB / 600M rows, Parquet, partitioned by date
Production benchmarks
5.5 TB across 10 production tables
Real workloads. Real data. Batch, streaming, delete-heavy, and multi-writer tables — same engine, same hardware.
| Table | Size | Workload | Files (B → A) | Throughput | Time |
|---|---|---|---|---|---|
| balance_snapshots | 1,192 GB | TB-Scale batch | 11,957 → 3,270 | 1,572 MB/s | 11 min |
| events_analytics | 484 GB | Delete-Heavy | 16,128 → 7,198 | 729 MB/s | 11m 21s |
| raw_sdk_events | 8 GB | Streaming | 42,633 → 69 | 167 MB/s | 138s |
| site_traffic | 292 GB | Multi-Writer | 2,740 → 754 | 1,465 MB/s | 3m 25s |
Compaction speed
200 GB benchmark (seconds)
95% faster
Cost per TB
Normalized to Spark = 100%
90% cheaper
Query time
Avg. query latency after compaction
8x faster queries
Beyond file compaction
Full metadata optimization in one pass
Every compaction cycle also optimizes manifests, applies delete files, and computes statistics — so query planning and execution stay fast at any scale.
Rewrite Manifests
200+ → ~30, faster planning
Apply Deletes
Zero read-time reconciliation
Puffin Statistics
Aggressive file-level pruning
Expire Snapshots
Reclaim orphaned storage
Comparison
Why not just use Spark?
Spark was built for batch ETL, not continuous table maintenance. It works — but at 7× the time, 10× the cost, and with operational complexity that blocks continuous optimization.
| Spark | AWS S3 Tables | LakeOps | |
|---|---|---|---|
| 200 GB compaction | 1,612s | 6,300s | 221s |
| Peak throughput | ~350 MB/s | ~32 MB/s | 2,522 MB/s |
| Memory model | JVM heap + GC | Managed | Bounded (no OOM) |
| Query-aware sort | Manual config | No | Automatic |
| Trigger model | Cron / manual | Background | Events + policies |
| Infrastructure | Cluster required | Managed | Serverless |
Benchmark: 200 GB / 600M rows, Parquet, partitioned by date. Same hardware, same data, same target file size.
Part of the platform
Compaction in context
In LakeOps, compaction is one step in a coordinated maintenance pipeline — automatically paired with observability, governance, query routing, and cost optimization in a single control plane.
Compaction
Observability
Governance
Query Routing
Cost Optimization
AI Readiness
Resources
Go deeper
Optimizing Iceberg Lakehouse Performance
Six layers that determine query speed and how to optimize each one.
Iceberg Lake Compaction: A Guide
Event-driven triggers, query-aware sort, Rust engine, and coordinated maintenance.
Iceberg Lakehouse Observability
Table health, metrics, and cross-engine telemetry in one control plane.
Get started
See compaction in action
Connect your catalog and get a free compaction analysis in 10 minutes — see exactly which tables need attention, how much compute you can save, and what query speedup looks like on your workload.
No commitment · Typically 30 min