Replace Spark for Iceberg Compaction: Faster, Cheaper Alternatives

If you've ever been paged at 3 AM because a Spark compaction job OOM'd on a 1.2 TB partition, or watched a 45-minute EMR cluster spin up just to merge 200 small files, you already know the problem. Spark compaction works — until it doesn't. And when it fails, it fails expensively.

The economics are brutal. At roughly $50/TB in compute cost, Spark-based compaction across a production lakehouse with hundreds of tables and daily maintenance runs becomes one of your largest recurring infrastructure bills. A single 200 GB binpack job costs $1.54 in compute and takes 1,612 seconds to complete. Multiply that by 50–100 tables with nightly schedules, and you're spending more on keeping your lake healthy than on querying it.

The architectural mismatch is even worse. Compaction is fundamentally a narrow, I/O-bound read-merge-write operation with predictable memory requirements and no complex computation. Running it on a general-purpose distributed computation engine designed for arbitrary DAGs is like hiring a construction crew to rearrange your bookshelf. This article covers two paths forward: replacing Spark entirely with a purpose-built engine like LakeOps — which processes the same data at $5/TB and completes in 221 seconds what Spark takes 1,612 seconds to do — or optimizing your existing setup with lighter alternatives.

Why Spark is the wrong tool for compaction

Spark was built to process complex DAGs — multi-stage transformations, joins, aggregations, and shuffles across heterogeneous workloads. Compaction needs none of that. It reads Parquet files, optionally sorts rows, and writes new Parquet files. The small files problem is real and pervasive, but Spark is an over-engineered solution for it. Here's why the mismatch costs you:

JVM overhead and garbage collection

Every Spark executor runs on the JVM. GC pauses stall processing at unpredictable intervals. For a sustained I/O operation like compaction, these pauses interrupt what should be continuous sequential reads and writes. The JVM also forces you to choose between heap-heavy configurations (more GC pauses) or off-heap allocations (more complex memory management, risk of native OOM). There's no configuration that eliminates this — it's structural to the runtime.

Distributed shuffle for local operations

Spark's execution model assumes data needs to be shuffled between nodes. Compaction doesn't need a shuffle — it reads files from object storage and writes files back to object storage. The coordination overhead of task scheduling, executor provisioning, and driver-executor communication adds latency to an operation that could run as a single-process pipeline reading and writing to S3.

Cluster provisioning cost and time

Before your compaction job processes a single byte, you're paying for EMR cluster startup (2–8 minutes), Spark context initialization, catalog connection setup, and executor registration. On EMR, you're also paying for the master node that sits idle during data processing. Dynamic allocation helps but adds its own latency as executors scale up and down.

OOM failures on large partitions

This is the most common production failure. A real example from a GitHub issue: a user running Spark 3.5.3 on EMR 7.7.0 with 30 m5.8xlarge nodes and 96 GB executor memory could not compact an 800 GB unpartitioned table with 100K+ data files. Even reducing max-concurrent-file-group-rewrites to 1 and max-file-group-size-bytes to 5 GB, tasks still failed with exit code 137 (OOM killed by YARN). The typical workaround is splitting the job across partition filters and hoping each slice fits in memory — essentially writing a distributed scheduler on top of a distributed scheduler.

Here's what that defensive Spark compaction configuration looks like in practice:

1from pyspark.sql import SparkSession2 3spark = SparkSession.builder \4    .config("spark.driver.memory", "96g") \5    .config("spark.executor.memory", "96g") \6    .config("spark.executor.memoryOverhead", "8g") \7    .config("spark.sql.catalog.glue_catalog", "org.apache.iceberg.spark.SparkCatalog") \8    .config("spark.sql.catalog.glue_catalog.catalog-impl", "org.apache.iceberg.aws.glue.GlueCatalog") \9    .config("spark.sql.catalog.glue_catalog.io-impl", "org.apache.iceberg.aws.s3.S3FileIO") \10    .getOrCreate()11 12COMPACTION_OPTIONS = {13    'target-file-size-bytes': '536870912',          # 512 MB14    'max-file-group-size-bytes': '5368709120',      # 5 GB — keep small to avoid OOM15    'partial-progress.enabled': 'true',16    'partial-progress.max-commits': '10',17    'min-input-files': '5',18    'max-concurrent-file-group-rewrites': '1',      # Serialize to stay in memory19    'rewrite-all': 'true',20}21 22partitions = spark.sql("""23    SELECT DISTINCT dt FROM glue_catalog.db.events24    WHERE dt >= '2026-07-01'25""").collect()26 27for row in partitions:28    try:29        spark.sql(f"""30            CALL glue_catalog.system.rewrite_data_files(31                table => 'db.events',32                strategy => 'sort',33                sort_order => 'event_time ASC NULLS LAST',34                where => 'dt = \"{row.dt}\"',35                options => map(36                    'target-file-size-bytes', '536870912',37                    'max-file-group-size-bytes', '5368709120',38                    'partial-progress.enabled', 'true',39                    'max-concurrent-file-group-rewrites', '1'40                )41            )42        """)43    except Exception as e:44        print(f"Failed on partition {row.dt}: {e}")45        continue

That's 50+ lines of defensive configuration, manual partition iteration, and error swallowing — for a single table. Now multiply that by every table in your lake.

Fixed scheduling wastes compute

Cron-based compaction runs whether the table needs it or not. A table with 3 new files since yesterday gets the same 45-minute Spark job as a streaming table that accumulated 40,000 files. Over-compaction wastes money; under-compaction lets hot tables degrade. Neither outcome is acceptable at scale. For a deep dive into how compaction strategy choice compounds this problem, see Iceberg Compaction Strategies: Binpack, Sort, and Z-Order.

No conflict awareness

Spark compaction jobs don't know about concurrent writers. A compaction that takes 25 minutes to complete creates a 25-minute window where any concurrent write can cause a commit conflict — and the entire compaction job retries from scratch. On high-throughput streaming tables, this creates a feedback loop: long compaction → conflicts → retries → even longer compaction.

The hidden cost: operational toil

Beyond compute dollars, Spark compaction imposes ongoing engineering burden. Someone has to maintain the Airflow DAGs (or Step Functions, or cron jobs). Someone has to tune executor memory per table as data volumes grow. Someone has to handle alerts when jobs fail at 3 AM — and those failures are frequent. A Cloudera engineering blog documents a table with 36,000 small files per file group where managing 5 concurrent groups meant Spark had to handle metadata for 180,000 files simultaneously — exceeding the 16 GB executor memory allocation and causing YARN to terminate containers. The fix? Manually tune max-concurrent-file-group-rewrites, max-file-group-size-bytes, and spark.executor.cores per table until you find a configuration that works. Then hope the data volume doesn't change.

This is not a configuration problem with a configuration solution. It's an architectural mismatch between a general-purpose distributed engine and a narrow I/O-bound operation. The question is what replaces it.

Two paths forward

There are two strategies for escaping the Spark compaction trap. Path 1 replaces Spark entirely with a purpose-built compaction engine — an intelligent, continuously optimizing system that handles scheduling, execution, and observability autonomously. Path 2 is the manual route: tune your existing Spark setup, or adopt lighter tools like Athena OPTIMIZE, AWS Glue, or custom DataFusion scripts. Each has trade-offs. We'll start with Path 1 because the economics are decisive.

Path 1: Intelligent continuous optimization with LakeOps

LakeOps is an autonomous control plane for Apache Iceberg table maintenance. Rather than replacing one compaction script with another, it replaces the entire maintenance workflow — scheduling, execution, monitoring, and optimization — with a system that runs continuously and improves over time.

At its core is a purpose-built compaction engine written in Rust and powered by Apache DataFusion. It was designed from scratch for the specific problem of Iceberg file optimization: reading Parquet, merging or sorting rows, and writing new Parquet files back to object storage. No JVM. No distributed scheduler. No cluster provisioning. The engine connects to your existing catalog (Glue, Hive, REST, Polaris) and handles everything from there.

Here's what that means in practice — and why the architecture produces fundamentally different results than Spark.

The Rust + DataFusion engine

The compaction engine processes Parquet data through Arrow columnar buffers with:

• Zero GC: No JVM, no garbage collection pauses, no unpredictable stalls
• Bounded memory: Spills to disk gracefully — impossible to OOM regardless of table size
• Lock-free parallelism: Worker threads never stall waiting on each other
• Zero-copy Arrow format: No serialization between processing stages
• No cluster provisioning: Single-process execution, no executor coordination overhead

The memory model is the critical difference. Where Spark requires you to over-provision (and still OOMs), LakeOps uses bounded memory per worker with disk spill as backpressure. A 1.192 TB table that caused Spark to OOM on identical hardware completed in 11 minutes at 1,572 MB/s throughput. For a full comparison of how this engine stacks up against every alternative, see 9 Iceberg Compaction Tools Compared.

Production benchmarks

These numbers come from published benchmarks across 10 real production tables totaling 5.5 TB:

Per-table results across production workloads:

The overall result: 101,223 → 19,170 files (81% reduction) across 5.5 TB. On that 1.2 TB table where Spark OOM'd, LakeOps finished in 11 minutes. A streaming table with 42,633 small files was compacted to 69 files — 99.8% reduction — in 138 seconds.

Self-improving execution

The engine records per-table throughput, partition structure, and memory usage from each run. Subsequent passes execute faster as the planner converges on optimal resource allocation. In production, the same 1.192 TB table went from 22 minutes to 11 minutes across consecutive runs spanning 2 days — a 70% throughput jump from 925 MB/s to 1,572 MB/s with zero configuration changes.

This matters because lakehouse data doesn't arrive once — it arrives continuously. A compaction engine that gets faster over time means your maintenance costs trend downward while your data volume trends upward. Spark's cost scales linearly with data; LakeOps's cost curves downward as the planner learns workload patterns.

Conflict-aware compaction

Because the Rust engine completes compaction in minutes rather than tens of minutes, the commit conflict window shrinks dramatically. A 221-second compaction has a fraction of the conflict probability compared to a 1,612-second Spark job. Additionally, LakeOps coordinates compaction safely around concurrent writers — it understands which files are being written to and avoids rewriting them. For streaming tables with continuous ingestion, the difference between a 3-minute maintenance window and a 25-minute window is the difference between seamless background optimization and a constant stream of commit conflicts.

Adaptive triggers replace cron schedules

LakeOps continuously monitors structural signals per table — file count, average file size relative to target, delete-file-to-data-file ratio, manifest depth. Compaction fires when table health degrades, not on a fixed timer. A streaming table ingesting files every 60 seconds might compact multiple times per hour. A weekly batch table compacts once after its load completes. No wasted runs on idle tables, no missed maintenance on hot tables. This is the core of what automating Iceberg table maintenance should look like — reactive to actual table state, not a cron schedule someone set six months ago.

Query-aware sort optimization

The engine collects telemetry from connected query engines (Trino, Spark, Snowflake, Athena, DuckDB, Flink) and identifies which columns appear in WHERE, JOIN, and GROUP BY clauses per table. Sort compaction then physically reorganizes data to maximize data skipping for actual production queries — not static sort orders guessed at table creation time. This delivers up to 12x query performance improvement on sorted tables.

Lake-wide policies

Compaction rules are declared as policies with a specificity hierarchy: table-level overrides namespace, which overrides catalog-wide baseline. One policy definition replaces dozens of per-table Spark DAGs. Platform teams set lake-wide defaults; individual teams override for their SLAs.

What LakeOps replaces

With LakeOps handling compaction autonomously, you eliminate:

• EMR/Dataproc clusters dedicated to compaction
• Airflow DAGs managing per-table Spark jobs
• Custom partition-iteration scripts to avoid OOM
• On-call rotations for failed compaction jobs
• Over-provisioned executor memory "just in case"
• Manual sort order configuration per table

Watch demo·LakeOps product demoYouTube ↗

Path 2: The manual/DIY path

If you're not ready to replace Spark entirely, there are intermediate steps — from optimizing your existing setup to using cloud-native tools that trade flexibility for simplicity. Each option addresses some of Spark's problems while inheriting others.

Optimizing existing Spark compaction

If Spark must stay, these configuration changes address the most common failure modes. Note that these are mitigations, not solutions — they reduce failure frequency without addressing the root cause:

1. Enable partial progress — commit incrementally so a late-stage OOM doesn't lose all work:

1'partial-progress.enabled': 'true'2'partial-progress.max-commits': '10'

2. Reduce concurrent file group rewrites — serialize processing to stay within memory bounds:

1'max-concurrent-file-group-rewrites': '1'

3. Limit file group size — keep individual tasks small enough to fit in executor memory:

1'max-file-group-size-bytes': '5368709120'  # 5 GB

4. Filter by partition — compact specific date ranges rather than the entire table:

1CALL catalog.system.rewrite_data_files(2    table => 'db.events',3    where => 'dt >= "2026-07-01" AND dt < "2026-07-08"'4)

5. Increase driver memory for metadata planning — the driver scans all manifests to plan file groups:

1spark.driver.memory=16g

These help, but they don't change the fundamental cost structure. You're still paying for JVM overhead, cluster provisioning, and over-provisioned executors on every run. And every one of these settings represents a future failure waiting to happen — the moment your data volume exceeds the assumptions baked into these numbers, you're back to debugging OOMs.

Athena OPTIMIZE

AWS Athena offers a single SQL statement for Iceberg compaction:

1OPTIMIZE my_table REWRITE DATA USING BIN_PACK2WHERE dt >= '2026-07-01'

Advantages:

• No infrastructure to manage — serverless
• Simple SQL interface — no Spark configuration
• Integrated with Glue Catalog

Limitations:

• Binpack only — no sort or z-order strategy, so no data skipping improvement
• 100 partition limit per execution — must loop with WHERE clauses for larger tables
• Only partition columns in WHERE clause — can't target specific file conditions
• 20,000 object limit on VACUUM operations
• Resource exhaustion on tables with heavy delete file accumulation
• Cost scales with data scanned — charged per byte read during rewrite

Athena OPTIMIZE works for small tables with simple binpack needs. For anything requiring sort optimization, handling more than 100 partitions, or processing delete-heavy tables, you'll hit its limits quickly.

AWS Glue compaction jobs

AWS Glue provides a managed Spark environment with DPU-based pricing. It's the same Spark engine underneath but removes cluster management:

1import sys2from awsglue.utils import getResolvedOptions3from pyspark.sql import SparkSession4 5spark = SparkSession.builder \6    .config("spark.sql.catalog.glue", "org.apache.iceberg.spark.SparkCatalog") \7    .config("spark.sql.catalog.glue.catalog-impl", "org.apache.iceberg.aws.glue.GlueCatalog") \8    .getOrCreate()9 10spark.sql("""11    CALL glue.system.rewrite_data_files(12        table => 'db.events',13        strategy => 'sort',14        sort_order => 'event_time ASC'15    )16""")

Advantages:

• No EMR cluster management
• Supports sort and z-order strategies via rewrite_data_files
• Auto-scaling workers
• Pay per DPU-hour

Limitations:

• Still JVM/Spark under the hood — same OOM risks on large tables
• DPU pricing can be expensive at scale (~$0.44/DPU-hour)
• Cold start latency (1–3 minutes)
• Still requires per-table job configuration
• No adaptive triggers — you schedule it yourself

Custom DataFusion scripts

For teams comfortable writing Rust or Python, Apache DataFusion provides the building blocks for a custom compaction engine:

1use datafusion::prelude::*;2use iceberg::table::Table;3 4async fn compact_partition(table: &Table, partition: &str) -> Result<()> {5    let ctx = SessionContext::new();6    let df = ctx.read_parquet(partition_path, ParquetReadOptions::default()).await?;7    let sorted = df.sort(vec![col("event_time").sort(true, true)])?;8    sorted.write_parquet(output_path, None).await?;9    Ok(())10}

Advantages:

• Same DataFusion engine LakeOps uses — bounded memory, Arrow-native
• Full control over file group planning and output sizing
• No JVM overhead
• Can be deployed as a lightweight container

Limitations:

• You own the entire compaction lifecycle: file planning, conflict handling, commit coordination
• No adaptive triggers — you build the scheduling
• No cross-engine telemetry for sort optimization
• No lake-wide policies or observability
• Significant engineering investment to handle edge cases (partial failures, concurrent writers, manifest rewrites)

This path makes sense if you have a single large table with specific compaction needs and strong Rust/DataFusion expertise. It doesn't scale to managing hundreds of tables.

Dremio Arctic / Nessie maintenance

Dremio's lakehouse platform includes table optimization capabilities through its Arctic catalog (built on Project Nessie):

1OPTIMIZE TABLE my_table2REWRITE DATA USING BIN_PACK

Advantages:

• Integrated with Dremio's query engine
• Supports bin-pack and sort strategies
• Branch-level maintenance (compact on a branch, merge when verified)

Limitations:

• Tied to Dremio's ecosystem — requires Nessie/Arctic catalog
• Still JVM-based execution
• Limited to Dremio-managed tables
• No cross-engine query telemetry for sort optimization

Cost comparison at scale

For a production lakehouse compacting 10 TB/day across 200 tables:

The cost gap compounds over time. Spark costs remain linear with data volume. LakeOps costs grow sublinearly because the self-improving planner and adaptive triggers eliminate redundant compaction runs on tables that don't need them. One team documented cutting their compaction bill from $8,400/year to $750 on a single table — roughly a 90% reduction that compounds across every table in the lake.

Beyond raw compute, consider the engineering hours. A platform team maintaining Spark compaction across 200 tables typically spends 10–15 hours/week on maintenance: tuning configurations, responding to OOM alerts, investigating slow queries caused by compaction backlogs, and updating DAGs as tables evolve. That's a $100K+/year fully-loaded engineering cost that doesn't show up in your AWS bill.

Migration path

Replacing Spark compaction doesn't require a big-bang migration. Compaction is a background maintenance operation — you can swap the engine without touching any data pipelines, query patterns, or table schemas.

Week 1: Connect LakeOps to your existing catalog (Glue, Hive, REST). It reads metadata only — no data access required for initial assessment. Within minutes you'll see which tables are fragmented and which are healthy.

Week 2: Run LakeOps compaction on 2–3 non-critical tables alongside your existing Spark jobs. Compare duration, cost, and resulting file layout. Production benchmarks consistently show 86–95% improvement in speed and 80–90% reduction in cost.

Week 3–4: Disable Spark compaction DAGs for migrated tables. Set lake-wide policies in LakeOps. The adaptive trigger system handles scheduling automatically based on table health signals.

Week 5+: Decommission dedicated compaction EMR clusters. Redirect the budget to actual compute workloads that benefit from Spark's strengths — complex transformations, ML pipelines, interactive analytics. Spark remains excellent for these workloads. It just shouldn't be running compaction.

Conclusion

Spark is an extraordinary tool for distributed computation. It's the wrong tool for compaction. The JVM overhead, cluster provisioning, OOM fragility, and fixed scheduling make it structurally expensive for an operation that should be fast, cheap, and invisible.

The evidence is unambiguous: LakeOps's purpose-built Rust engine processes the same data 7–8x faster at 10% of the cost, handles tables that cause Spark to OOM, and improves with every run. The architectural advantages — bounded memory, zero GC, lock-free parallelism — aren't marginal optimizations. They're structural differences that make Spark's failure modes impossible.

If you're spending $50/TB on compaction, getting paged for OOM failures, or maintaining Airflow DAGs just to keep your tables healthy — the problem isn't your configuration. It's the architecture. Purpose-built compaction eliminates these problems structurally, not through better tuning.

Either way, stop paying $50/TB for what should cost $5.

---

Further reading:

• Iceberg Compaction Strategies: Binpack, Sort, and Z-Order — deep dive into when each strategy applies
• 9 Iceberg Compaction Tools Compared — full feature matrix across all production options
• Automating Iceberg Table Maintenance — beyond compaction: snapshot expiration, orphan cleanup, manifest optimization
• The Small Files Problem in Apache Iceberg — root causes and prevention strategies
• State of Iceberg FinOps: Cost Reduction — where compaction cost fits in the broader optimization picture