Efficient Compaction
Efficient Compaction for
Apache Iceberg
LakeOps's purpose-built Rust engine compacts Iceberg tables 95% faster than Spark — query-aware data sorting, bounded memory, and continuous optimization keep every table lean and fast without manual intervention.
The Engine
Built on Rust & Apache DataFusion
No JVM. No garbage collection pauses. No OOM crashes. LakeOps compaction is a purpose-built Rust binary powered by Apache DataFusion — the same query engine behind Apache Arrow and InfluxDB 3.0.
Zero-copy memory
Arrow columnar format with no serialization overhead between read, sort, and write stages.
Bounded memory
Spills to disk gracefully — compacts TB-scale tables on a single 8 GB node without OOM risk.
Native Parquet I/O
Reads and writes Parquet natively with predicate pushdown and projection — no unnecessary data copied.
Comparison
Why not just use Spark?
Spark was built for batch ETL, not continuous table maintenance. It works — but at 7× the time, 10× the cost, and with operational complexity that blocks teams from running compaction continuously.
| Spark | AWS S3 Tables | LakeOps | |
|---|---|---|---|
| 200 GB benchmark | 1,612s | 6,300s | 221s |
| Peak throughput | ~350 MB/s | ~32 MB/s | 2,522 MB/s |
| Memory model | JVM heap + GC | Managed | Bounded (no OOM) |
| Query-aware sort | Manual config | No | Automatic |
| Infrastructure | Cluster required | Managed | Serverless |
| Cost per TB | ~$40–80 | ~$15 | $5 |
Benchmark: 200 GB / 600M rows, Parquet, partitioned by date. Same hardware, same data, same target file size.
The problem
Why Iceberg tables accumulate technical debt
Iceberg's metadata architecture is built for fast queries. But without active maintenance, physical table state degrades — and every query pays the penalty.
Small files multiply per-query overhead
Streaming ingestion creates thousands of tiny files. Each file costs an S3 GET request, a metadata read, and a connection — query time scales with file count, not data volume.
Unsorted data defeats data skipping
Without sort order aligned to query patterns, Parquet min/max statistics are useless. Engines scan every row group regardless of predicate filters.
Fragmented manifests bloat planning time
Hundreds of small manifests force the query planner to read excessive metadata. Planning often dominates total query time at 200+ manifests per table.
Delete files compound read amplification
Merge-on-read tables accumulate position delete files. Every query reconciles deletes at read time — performance degrades linearly with delete file count.
Results
Measured impact on
real Iceberg workloads
Benchmarks from production-grade tables across multiple engines and cloud providers.
Compaction speed
vs. Apache Spark on identical datasets
Query performance
After compaction + layout optimization
Cost savings
In compute & storage spend
Table health
Autonomous maintenance keeps every table optimized
How LakeOps Compaction Works
Four layers of
intelligent compaction.
Query-aware sorting + Rust speed + metadata optimization + delete cleanup = continuously optimized tables.
Query-aware compaction
Data sorted by how it's actually queried
LakeOps tracks which columns appear in WHERE, JOIN, and GROUP BY clauses for every table. During compaction, data is physically sorted by those columns — so Parquet row group statistics enable engines to skip irrelevant data without reading it.
- 51% less data scanned — sorted by real filter columns, per table
- 47,000 → 280 files: same data, same query — 52s drops to 5.8s
- Self-improving: sort strategy adapts as query patterns evolve
Files before
47,000
Files after
280
Query before
52s
Query after
5.8s
Scan volume reduced 51%
Query-aware sort + optimized file layout
95% faster Rust engine
Tables stay optimized because compaction is fast enough to run continuously
A purpose-built Rust engine with Apache DataFusion eliminates JVM/GC overhead. Compaction completes in minutes instead of hours — so tables never degrade between maintenance windows.
- 221s vs 1,612s (Spark) vs 6,300s (S3 Tables) on identical 200 GB
- 2,522 MB/s peak throughput — TB-scale tables compacted in minutes
- Bounded memory: no OOM crashes regardless of table size
Manifest & metadata optimization
Query planning stays fast at any table scale
LakeOps consolidates fragmented manifests and computes Puffin column statistics (NDV, min/max, null counts). Planners read fewer manifests and make smarter skip decisions — planning drops from seconds to milliseconds.
- Manifest consolidation: 200+ manifests → ~30 in a single atomic rewrite
- Puffin statistics enable aggressive file-level pruning across all engines
- Auto-triggered after compaction cycles — manifests never drift
Rewrite Manifests
Consolidate for faster planning
Rewrite Position Deletes
Eliminate read-time overhead
Compute Puffin Statistics
Enable aggressive file pruning
Delete file optimization
Eliminate read-time reconciliation overhead
Position delete files from merge-on-read workloads accumulate and force every query to reconcile deletions at scan time. LakeOps consolidates and physically applies delete files so reads are always clean.
- Rewrite Position Deletes: consolidate without full table rewrite
- Full compaction: physically merge deletes — zero read-time overhead
- 23,433 delete files (551M rows) cleaned in one compaction cycle
Delete files
23,433
After cleanup
0
Rows affected
551M
Read overhead
Eliminated
Runs on your stack
Resources
Go deeper
Optimizing Iceberg Lakehouse Performance
Six layers that determine query speed and how to optimize each one.
Optimizing Iceberg Lake Compaction: A Guide
Event-driven triggers, query-aware sort, Rust engine, and coordinated maintenance.
Iceberg Lakehouse Observability
Table health, insights, metrics, and cross-engine telemetry in one control plane.
Minutes to value with no risk
Connect & collect telemetry
Manual or autonomous management
Operations run & optimize
Observability & governance
Production benchmarks
5.5 TB across 10 production tables
Real workloads. Real data. Batch, streaming, delete-heavy, multi-writer, and terabyte-scale tables — all on the same engine, same hardware.
| Table | Size | Workload | Files (B → A) | Throughput | Time | Notes |
|---|---|---|---|---|---|---|
| balance_snapshots | 1,192 GB | TB-Scale batch | 11,957 → 3,270 | 1,572 MB/s | 11 min | Spark OOM on same hardware |
| user_accounts | 174 GB | Batch | 878 → 400 | 2,269 MB/s | 74s | Single Node |
| events_analytics | 484 GB | Delete-Heavy | 16,128 → 7,198 | 729 MB/s | 11m 21s | 23,433 delete files; 551M rows removed |
| raw_sdk_events | 8 GB | Streaming | 42,633 → 69 | 167 MB/s | 138s | 99.8% file reduction |
| site_traffic | 292 GB | Multi-Writer | 2,740 → 754 | 1,465 MB/s | 3m 25s | Single partition |
| cluster_registry | 322 GB | Batch | 998 → 440 | 2,522 MB/s | 2m | Peak throughput |
Compaction cost per TB
Normalized to Spark = 100%
Source: 200 GB (~1 TB uncompressed) benchmark. Spark cost index 100 vs LakeOps 10.
Self-improving: same table, zero config changes
balance_snapshots — 1.192 TB across consecutive runs
Same data and hardware; planner learns workload telemetry and improves runtime from 22 to 11 minutes.
Part of the platform
Compaction doesn't run in isolation
In LakeOps, compaction is one step in a coordinated maintenance sequence — automatically paired with snapshot expiration, orphan cleanup, manifest optimization, and Puffin statistics in a single adaptive policy.
Compact
Merge small files, sort by query patterns
Expire Snapshots
Remove outdated metadata and data
Rewrite Manifests
Consolidate for faster planning
Orphan Cleanup
Delete unreferenced storage objects
Get started
See compaction in action
Connect your catalog and get a free compaction analysis in 10 minutes — see exactly which tables need attention, how many files can be eliminated, and what cost savings look like on your workload.
No commitment · Typically 30 min
Lakehouse Control Plane
Compaction is one piece. See the full platform.
Explore maintenance, routing, observability, governance, and AI readiness alongside 95% faster compaction.
Managed Lakehouse
The complete control plane — compaction, maintenance, cost optimization, routing, observability, governance, and AI readiness.
Explore platformIceberg Maintenance
Snapshot expiration, manifest rewrites, orphan cleanup, and metadata — automated, sequenced, and safe.
Explore maintenance95% Faster Compaction
Rust engine, not Spark. Sorts data by real query patterns so reads skip entire file groups. $5/TB vs $50.
Explore compactionQuery Routing & Acceleration
Route queries across Trino, Spark, Snowflake, and more — optimized for cost, latency, or throughput per workload.
Explore routingObservability & Governance
Table health, insights, cross-engine telemetry, policies, retention, and audit trails — one control plane.
Explore observabilityAgentic AI Readiness
Agent-native MCP interface, guardrails, and a self-optimizing lake ready for AI agents and autonomous pipelines.
Explore AI enablement