Smart Compaction
Watches production WHERE, JOIN, and GROUP BY columns, then rewrites files to match. Engines skip more data. Runs 95% faster than Spark compaction on the same tables.
LakeOps| Operation | Table | Duration | Impact | Time | Status |
|---|---|---|---|---|---|
| Compact Data Files | customer_orders orders | 4s | 1.24 TB, 16 → 1 files | 57 minutes ago | SUCCESS |
| Expire Snapshots | payment_transactions payments | 27s | 8.2 TB | 4 hours ago | SUCCESS |
| Expire Snapshots | inventory_snapshots_20250702 warehouse | 3s | 2.1 TB | 4 hours ago | SUCCESS |
| Rewrite Manifests | raw_clickstream analytics | 1.9s | 3 → 1 manifests | 5 hours ago | SUCCESS |
| Compact Data Files | product_catalog products | 6m 11.3s | 3,008 → 1,256 files | 6 hours ago | SUCCESS |
Open and runs on your stack
How it works
Not just file merging — a continuous, query-informed optimization loop running on a Rust engine fast enough to run between writes.
Monitoring started at seq #4,812. Everything written from that point on is in scope.
Adaptive Intelligence
LakeOps continuously monitors every table: file count, average file size vs target, small-file ratio, delete-file accumulation, manifest depth, and write velocity. Each table is classified as Healthy, Warning, or Critical. Compaction fires only when a table crosses a health threshold — and the most degraded tables run first. The full maintenance pipeline is sequenced automatically: snapshot expiry before compaction (so you never rewrite files about to be GC’d), then orphan cleanup, then manifest rewrite.
Binpack compaction · 200 GB / 600M rows
Production tables
balance_snapshots1,192 GB11,957\u21923,270raw_sdk_events8 GB42,633\u219269cluster_registry322 GB878\u219240095%
Faster
90%
Cheaper
0
OOM risk
0
Clusters
Rust + DataFusion
Compaction is a narrow, I/O-bound read-merge-write operation. Spark runs it on a general-purpose distributed engine with JVM overhead, GC pauses, and OOM risk. LakeOps replaces that with a purpose-built Rust engine powered by Apache DataFusion: zero-copy Arrow columnar pipeline, bounded memory with disk spill, lock-free parallelism — no cluster to provision, no executor to tune. The engine learns from each run, so consecutive passes get faster without any configuration changes.
customer_idevent_dateproduct_idregionBefore
After
12×
Faster queries
76%
Less CPU
95%
vs Spark
90%
Cheaper ops
Query-Aware Sort
Most compaction tools just merge small files. LakeOps collects WHERE, JOIN, and GROUP BY column frequency from every connected engine — Trino, Spark, Snowflake, Athena, DuckDB — then physically re-sorts data files so Parquet min/max pruning skips entire file groups. The sort order adapts as query patterns evolve.
Sort by event_date, regionsim/sort-date-region
Sort by customer_id, event_dateBest matchsim/sort-cust-date
Sort by product_id, regionsim/sort-prod-region
76%
Scan reduction
12.4×
Query speedup
0
Production risk
Layout Simulations
Not sure whether sorting by event_date or customer_id helps more? LakeOps runs layout simulations on Iceberg branches — testing sort orders against real query patterns without touching production. Compare scan reduction, file layout, and estimated speedup side by side, then apply the winner.
Before
After
Delete Resolution
Merge-on-read tables accumulate position and equality delete files that engines must reconcile at query time. LakeOps physically applies deletes during the compaction pass — eliminating delete files, reclaiming storage, and restoring read performance in one operation.
Benchmarks
Spark was built for batch ETL, not continuous table maintenance. Same data, same hardware — different results.
| Metric | Spark | S3 Tables | LakeOps |
|---|---|---|---|
| 200 GB compaction | 1,612s | 6,300s | 221s |
| Peak throughput | ~350 MB/s | ~32 MB/s | 2,522 MB/s |
| Memory model | JVM heap | Managed | Bounded (no OOM) |
| Query-aware sort | Manual | No | Automatic |
| Trigger model | Cron | Background | Events + policies |
| Infrastructure | Cluster | Managed | Serverless |
200 GB / 600M rows, Parquet, partitioned by date. Same hardware, same data, same target file size.
Results
Benchmarks from production-grade tables across multiple engines and clouds.
Query speed
After compaction + layout optimization
CPU reduction
Compute hours across all engines
Storage saved
Orphans, snapshots & bloat removed
Table health
Autonomous maintenance keeps every table optimized
Compaction Results
LakeOps Rust engine compacting batch, streaming, delete-heavy, and multi-writer Iceberg tables. Same hardware, same target file size — no Spark, no JVM, no OOM.
| Table | Size | Workload | Files (B → A) | Throughput | Time |
|---|---|---|---|---|---|
| balance_snapshots | 1,192 GB | TB-Scale batch | 11,957 → 3,270 | 1,572 MB/s | 11 min |
| events_analytics | 484 GB | Delete-Heavy | 16,128 → 7,198 | 729 MB/s | 11m 21s |
| raw_sdk_events | 8 GB | Streaming | 42,633 → 69 | 167 MB/s | 138s |
| site_traffic | 292 GB | Multi-Writer | 2,740 → 754 | 1,465 MB/s | 3m 25s |
200 GB benchmark (seconds)
95% faster
Normalized to Spark = 100%
90% cheaper
Avg. query latency after compaction
8x faster queries
Only metadata is processed — never retained or stored.
Telemetry reveals table health and actions needed.
Autopilot, manual approval, or policy-driven.
SOC 2, SSO, RBAC, dedicated support, and the scale your largest Iceberg lakes demand.
SOC 2 Type II, encryption, SSO/RBAC, and audit trails for regulated teams.
One control plane for your full lake. Real-time visibility, policies, and predictable performance.
Dedicated onboarding, training, and enterprise SLAs. Deploy in VPC or on-prem.
Resources
A deep guide to bin-pack, sort, and Z-order compaction strategies for Apache Iceberg — when to use each, how to configure them, and how to automate strategy selection across hundreds of tables.
Spark compaction is expensive, slow, and architecturally wrong for file rewrites. Here's how to replace it with purpose-built engines that cost 90% less and finish 95% faster.
Apache Iceberg ships the maintenance primitives but none of them run themselves. This guide covers why each operation matters, the correct execution order, and how to automate the full lifecycle.
Connect your catalog and get a free compaction analysis in 10 minutes.