Iceberg Lake Compaction Strategies: A Practical Guide

Every Iceberg tutorial eventually shows you rewrite_data_files. The advice sounds simple: pick a strategy, run it on a schedule, move on. That works for a single table you know well. It falls apart the moment you manage dozens — or hundreds — of tables, each with different write patterns, query shapes, and SLAs.

The real challenge is not running compaction. It is choosing the right strategy for each table, configuring it correctly, triggering it at the right time, and adapting when query patterns shift. This guide breaks down each strategy, the factors that drive selection, the parameters that matter, and a practical framework for applying the right strategy to every table — continuously and autonomously.

If you run LakeOps, this decision is handled for you: the platform observes query telemetry across all engines, evaluates table state, and applies the optimal strategy per table automatically. But understanding the mechanics is still valuable — it helps you validate what any system (manual or managed) is doing under the hood.

The three compaction strategies

Apache Iceberg supports three approaches to reorganizing data files. Each solves a different problem and carries different cost/performance trade-offs.

Bin-pack: consolidate without reordering

Bin-pack merges small files into target-sized files without sorting or reordering rows. It is the fastest and cheapest strategy — essentially a file-level concatenation that splits at the target size boundary.

1CALL catalog.system.rewrite_data_files(2  table => 'analytics.raw_events',3  strategy => 'binpack',4  options => map(5    'target-file-size-bytes', '536870912',6    'min-file-size-bytes', '67108864',7    'partial-progress.enabled', 'true'8  )9);

When to use it: High-frequency streaming ingest (Kafka, Flink, Spark Structured Streaming) where files arrive continuously at 1–50 MB. Bin-pack consolidates them into 256–512 MB files fast enough to run hourly without breaking the budget.

What it does not do: Improve data skipping or query-level pruning. Min/max column statistics in the output files reflect the random order data arrived — not how it gets queried. Bin-pack reduces file count overhead but does not make individual queries faster through better data layout.

When bin-pack is enough: For tables where queries always full-scan (ML training pipelines, batch ETL), or where data arrives pre-sorted (single-writer pipelines with natural ordering), bin-pack solves the small-file problem without the cost of sorting.

Sort: optimize for dominant query patterns

Sort compaction reads candidate files, globally sorts all records by one or more columns, and writes new files where each file covers a narrow, non-overlapping value range for the sort key.

1CALL catalog.system.rewrite_data_files(2  table => 'orders.customer_orders',3  strategy => 'sort',4  sort_order => 'order_date ASC, customer_id ASC',5  options => map(6    'target-file-size-bytes', '268435456',7    'partial-progress.enabled', 'true',8    'max-concurrent-file-group-rewrites', '5'9  )10);

The result: Parquet min/max statistics become highly selective. A query filtering WHERE order_date BETWEEN '2026-06-01' AND '2026-06-30' can skip 90%+ of files because each file covers only a small date range. This is data skipping — the single most impactful read optimization in Iceberg.

When to use it: Tables where 70%+ of queries filter or range-scan on the same 1–2 columns. Common patterns include time-series data filtered by timestamp, transactional data filtered by date or customer ID, and event tables filtered by event type.

Trade-off: Sort compaction has moderate-to-high write amplification. Every run rewrites all targeted data with a full distributed sort. Running it hourly is wasteful — daily or post-batch is the typical cadence.

The LakeOps difference: Rather than guessing which columns to sort by, LakeOps observes actual query patterns across all engines — which columns appear in WHERE clauses, with what frequency and selectivity — and selects the sort order that maximizes data skipping for your real workloads. When query patterns change (new dashboard, new team, new engine), the sort order adapts.

Z-order: multi-dimensional clustering

Z-order applies a space-filling curve (Z-curve) to interleave bits from multiple column values, clustering data across all specified columns simultaneously. Unlike linear sort — which strongly optimizes the first column and weakly optimizes subsequent ones — Z-order distributes clustering benefit evenly.

1CALL catalog.system.rewrite_data_files(2  table => 'analytics.ad_impressions',3  strategy => 'sort',4  sort_order => 'zorder(campaign_id, event_date, geo_region)'5);

When to use it: Ad-hoc analytical tables where queries filter on different column combinations unpredictably. If one analyst queries by campaign_id, another by geo_region, and a dashboard filters by event_date — Z-order ensures all three patterns benefit from file pruning.

Trade-off: Z-order is the most expensive strategy. The bit-interleaving computation is CPU-heavy, and clustering quality degrades above 4 columns. Limit Z-order to 2–4 high-cardinality, frequently-filtered columns. Run it weekly or during off-peak windows.

Hilbert curves (emerging): An alternative space-filling curve that avoids Z-order's quadrant-boundary discontinuities. Benchmarks show ~57% file skip rate vs Z-order's ~46% on the same dataset. Not yet in mainline Iceberg, but available via LakeOps's Rust engine for tables that justify the compute cost.

How LakeOps compacts: the Rust engine

Traditional compaction runs on Apache Spark — a general-purpose distributed compute framework. That means JVM startup overhead, garbage collection pauses, executor provisioning, shuffle stages, and idle cluster costs. For a maintenance operation that is fundamentally I/O-bound (read files, sort, write files), this is massive overkill.

LakeOps replaces Spark with a purpose-built Rust compaction engine based on Apache DataFusion. The key advantages:

• Zero-copy memory — Arrow columnar format with no serialization between stages
• Bounded memory — predictable resource usage, no GC pauses, no OOM kills
• Native Parquet I/O — direct column reads/writes without Java interop layers
• Streaming pipeline — read, transform, and write in a single pass without materializing intermediate results
• No cluster provisioning — single-process execution with internal parallelism, no executor coordination overhead

Benchmark results

On production workloads (5,515 GB across 10 real Iceberg tables): | Metric | Result | |---|---| | Files reduced | 101K → 19K (81% reduction) | | Peak throughput | 2,522 MB/s | | Max file reduction | 99.8% (42,633 → 69 files) | | Deleted rows cleaned | 551M rows | | Cost vs Spark | 10% (90% savings) | | Self-improvement | 22 min → 11 min over 3 runs | The engine handles streaming tables (raw_sdk_events: 42,633→69 files, 99.8% reduction), TB-scale batch tables (balance_snapshots: 1,192 GB in 11 min), delete-heavy tables (events_analytics: 23,433 delete files, 551M rows removed), and multi-writer tables — all on the same hardware where Spark OOMs.

Self-improving planner

LakeOps doesn't apply a static compaction plan. The planner learns from each run:

1. 1.First run: Baseline — reads table structure, applies configured strategy
2. 2.Second run: Learns file-size distribution and optimal group sizes from first run's output
3. 3.Third run (learned): Optimized parallelism, group sizing, and I/O scheduling based on actual table characteristics

On a 1.192 TB table: Run 1 took 22 min at 925 MB/s. Run 3 took 11 min at 1,572 MB/s — same data, same hardware, zero configuration changes.

Factors that determine the right strategy

No single strategy is universally best. The correct choice depends on measurable properties of each table:

1. Query patterns. Inspect the columns that appear in WHERE, JOIN, and GROUP BY clauses across all engines reading the table. If 70%+ of queries filter on the same 1–2 columns, sort wins. If filters spread across 3–4 columns unpredictably, Z-order wins. If there is no dominant filter pattern (or queries always full-scan), bin-pack is sufficient.

2. Write frequency. Streaming tables that receive commits every few minutes accumulate small files fast. They need frequent bin-pack (hourly) to stay readable, with periodic sort (daily) for layout optimization. Batch tables loaded once a day can go straight to sort or Z-order post-load.

3. Table size. Sort and Z-order only justify their cost above a certain data volume. On a 500 MB table, the performance difference is negligible. On a 500 GB table, proper sort order can cut query times from 52 seconds to under 6 seconds.

4. Partition scheme. If the table is already partitioned by date and most queries filter by date, the partition itself handles pruning — sort on a different column (e.g., customer_id) within each partition adds the most value. Compaction strategies and partition design are complementary, not redundant. For more on this relationship, see Iceberg Partitioning Strategies.

5. Engine diversity. When multiple engines (Trino, Spark, Snowflake, Athena) read the same table, their query patterns may diverge. A sort order optimized for Trino dashboards might not help Spark ETL. Z-order or a compromise sort key may be more appropriate. LakeOps aggregates cross-engine telemetry to make this decision automatically.

6. Delete file accumulation. On merge-on-read tables, positional and equality delete files pile up between compaction runs. Each pending delete file adds read-time merge overhead. Tables with high CDC velocity need aggressive compaction with low delete-file-threshold (3–5 files) to keep read performance stable.

Configuration parameters that matter

Target file size

The most impactful parameter. Determines the output file size after compaction.

Set via target-file-size-bytes in options or the table property write.target-file-size-bytes. LakeOps auto-tunes this based on observed partition sizes and query patterns — tables with point-lookup workloads get smaller targets, full-scan tables get larger targets.

Compaction thresholds

• `min-file-size-bytes` (default: 75% of target) — files below this size are candidates for rewrite.
• `max-file-size-bytes` (default: 180% of target) — oversized files may be split.
• `min-input-files` (default: 5) — minimum file count per group before compaction triggers. Prevents wasteful single-file rewrites.
• `delete-file-threshold` — number of delete files associated with a data file before it becomes a compaction candidate. Critical for merge-on-read tables.
• `max-file-group-size-bytes` (default: 100 GB) — bounds the maximum bytes in a single rewrite group. Breaks very large partitions into manageable chunks.
• `rewrite-job-order` — bytes-asc (smallest groups first, default), bytes-desc (largest first), or files-desc (most files first). Controls which groups get compacted first when time is limited.

Partial progress

On large tables, compaction jobs can run for hours. Without partial progress, a single Iceberg commit conflict invalidates the entire job — hours of compute wasted.

• `partial-progress.enabled: true` — commits rewritten file groups independently. If group 7 of 20 conflicts, only group 7 retries.
• `partial-progress.max-commits` (default: 10) — caps how many independent commits a single job creates.

Always enable partial progress on tables with concurrent writers or more than 10 partitions. LakeOps enables this by default for all tables.

Sort key selection

The sort key should reflect how data is read, not how it is written. The most common mistake is sorting by the ingestion timestamp because it is the write-time partition key — but if queries filter by user_id or event_type, sorting by those columns delivers far more skip benefit.

How to choose sort keys:

1. 1.Query your engine's query log for the top columns in WHERE clauses on this table.
2. 2.Rank by frequency × selectivity (a column filtered in 80% of queries that reduces scan by 10× beats one filtered in 20% of queries).
3. 3.Use the top 1–2 columns for sort, or top 2–4 for Z-order.
4. 4.Never duplicate the partition column in the sort order — within a day-partition, all event times already cluster within 24 hours.
5. 5.Consider column cardinality — low-cardinality sort keys (e.g., 5 event types) produce coarse file boundaries; combine with a higher-cardinality secondary key.

With LakeOps, this analysis happens automatically — cross-engine telemetry surfaces the real filter columns, and the platform simulates candidate sort orders before committing to one.

Production best practices

1. Layer strategies by cadence

The most effective production pattern combines strategies at different frequencies:

• Hourly: Bin-pack streaming partitions to consolidate small files (fast, cheap)
• Daily: Sort or Z-order on settled partitions where data is no longer actively written (expensive, high-value)
• Weekly: Full Z-order passes on ad-hoc analytical tables during off-peak windows

This layered approach keeps read performance high at every point in the data lifecycle without overspending on compaction compute.

2. Avoid compacting hot partitions

The most common cause of compaction commit conflicts. If a streaming writer is appending to today's partition while compaction rewrites files in the same partition, one of them will fail with a ValidationException.

Fix: Scope compaction to exclude actively-written partitions:

1CALL catalog.system.rewrite_data_files(2  table => 'events.page_views',3  strategy => 'binpack',4  where => 'event_date < current_date()'5);

LakeOps handles this automatically — it tracks which partitions have active writers and excludes them from compaction windows, eliminating conflict-related job failures entirely.

3. Sequence maintenance operations correctly

Compaction does not run in isolation. It is one step in a broader table maintenance lifecycle:

1. 1.Compact data files — merge and sort
2. 2.Rewrite manifests — consolidate metadata files
3. 3.Expire snapshots — remove old history (but keep enough for time travel)
4. 4.Remove orphan files — clean unreferenced objects left by failed writes

Order matters: compacting first ensures the snapshot metadata references the final file set. Expiring too soon risks removing files still needed by in-flight queries. Orphan cleanup should always lag behind other operations by 3+ days to avoid deleting files from in-flight jobs.

LakeOps executes this sequence automatically in the correct order, with each operation's output feeding the next — eliminating the redundant work that independently-scheduled scripts inevitably perform.

4. Use threshold-driven triggers, not fixed schedules

A nightly cron job wastes compute on tables that are already healthy and misses tables that degraded during the day. Effective production systems trigger compaction based on measurable conditions:

• Average file size drops below 64 MB
• File count per partition exceeds 100
• Delete file ratio exceeds threshold (merge-on-read tables)
• Query planning time degrades beyond baseline

LakeOps implements this as Adaptive Maintenance — each table is continuously evaluated and compaction triggers only when the table state warrants it. Tables under heavy write load get compacted hourly. Tables receiving one batch per day get compacted once post-load. Quiet tables aren't touched at all.

5. Dedicate compaction compute

Run compaction on a separate cluster or compute pool. Compaction is I/O intensive — it competes with analytical queries for object storage bandwidth. Mixing the two degrades both workloads.

The LakeOps Rust engine runs compaction 95% faster than Spark at $5/TB vs $50/TB, precisely because it uses purpose-built, bounded-memory compute that does not share resources with query engines. There is no cluster to provision, no executors to size, no GC to tune.

The compaction lifecycle in LakeOps

Understanding how LakeOps orchestrates compaction end-to-end:

1. Connect and observe

Connect your Iceberg catalog (AWS Glue, REST/Polaris, Nessie, S3 Tables). LakeOps begins collecting:

• Table-level metrics: file counts, sizes, partition distribution
• Query telemetry from every engine: filter columns, scan volumes, planning times
• Write patterns: commit frequency, file sizes, partition targets

No data moves. No pipelines change. Telemetry collection is metadata-only.

2. Analyze and recommend

For each table, the platform evaluates:

• Current health — file count, average file size, delete file ratio, manifest count
• Optimal strategy — bin-pack vs sort vs Z-order based on query patterns
• Sort order — which columns to sort by based on cross-engine filter frequency
• Cadence — how often to compact based on write velocity and degradation rate
• Layout simulations — projected query improvement for candidate sort orders

3. Execute autonomously

Compaction runs on the Rust engine with:

• Automatic hot-partition avoidance
• Partial progress enabled by default
• Conflict detection and automatic retry
• Priority scheduling (critical tables first)
• Self-improving planner that optimizes parallelism over consecutive runs

4. Measure and adapt

After each compaction:

• File reduction metrics are logged (before → after)
• Query planning time is compared pre/post
• Cost savings are calculated and attributed
• Strategy effectiveness is evaluated — if a sort order is not improving query times, the planner adjusts

Common anti-patterns

Running sort/Z-order too frequently. These strategies rewrite all targeted data every time. Hourly sort is pure waste — bin-pack hourly, sort daily.

Z-ordering on too many columns. Each additional column halves the clustering benefit per column. Beyond 4 columns, Z-order degrades to near-random ordering. Pick 2–4 high-cardinality, frequently-filtered columns.

Ignoring delete file accumulation. On merge-on-read tables, positional and equality delete files pile up between compaction runs. Every reader pays merge overhead on every pending delete file. Set delete-file-threshold aggressively (3–5 files) or use continuous compaction.

Global rewrites on multi-TB tables. Running rewrite_data_files without a where clause on a 10 TB table exhausts cluster resources and maximizes conflict windows. Always scope by partition range and use max-file-group-size-bytes to bound individual group sizes.

Multi-engine compaction conflicts. If both Spark and Trino attempt compaction on the same table, unpredictable conflicts result. Designate a single compaction owner — or use a control plane like LakeOps that coordinates across engines and tables.

Not enabling partial progress. Without it, a single conflict invalidates hours of compute. On a table with 100+ partitions, this means the entire job restarts from scratch. Always set partial-progress.enabled = true for large tables.

Using Spark for what should be a lightweight operation. Bin-pack compaction of a 50 GB table does not need a 10-node Spark cluster with 40 GB executors. It needs a single-process pipeline that reads and writes Parquet efficiently. Overprovisioned compaction clusters are one of the largest hidden costs in Iceberg operations.

Measuring compaction impact

How to know if your compaction strategy is actually working: | Metric | Before compaction | Healthy target | |---|---|---| | Avg file size | 5–50 MB (fragmented) | 128–512 MB | | Files per partition | 50–500+ | 1–20 | | Query planning time | 2–10s | < 500ms | | Data scanned per query | Full table | < 10% (with sort) | | Delete files pending | 50–1000+ | 0–5 | | Manifest count | 200–2000+ | 10–50 |

Automating strategy selection at scale

For teams managing more than a handful of tables, manual strategy selection does not scale. The factors that determine the right strategy — query patterns, write frequency, table size, engine diversity — change over time. A sort order that was optimal three months ago may be irrelevant after a product change shifts query patterns.

This is where a managed approach becomes essential. LakeOps continuously collects query telemetry from every engine reading each table, evaluates file-level health metrics, simulates candidate layouts, and applies the optimal strategy — adapting as workloads evolve. The result: every table gets the right compaction strategy, at the right time, without manual tuning or stale cron jobs.

The measured outcomes on production workloads:

• 95% faster compaction execution (Rust vs Spark)
• Up to 12× query performance improvement (sort-order optimization)
• 80% cost reduction (storage + compute + maintenance combined)
• Zero manual tuning — strategy selection, scheduling, and conflict management are autonomous

If you want to see what this looks like on your own tables — file counts, sort order recommendations, and projected query improvements — connect your catalog and get a free compaction analysis in under 10 minutes.

Further reading

• Optimizing Iceberg Lake Compaction — deep dive into event-driven compaction and the Rust engine architecture
• Autonomous Iceberg Table Maintenance — how compaction fits into the full maintenance lifecycle
• Iceberg Delete Files: Merge-on-Read Explained — why delete file handling matters for compaction strategy
• Iceberg Partitioning Strategies — how partition design complements sort order selection
• LakeOps Compaction Solution — benchmarks, architecture, and how it works