Airflow & Iceberg: Building Production Maintenance DAGs

Apache Airflow is the de facto orchestrator for data platform teams running Apache Iceberg. If your lakehouse runs on Spark, Trino, or Flink, your maintenance almost certainly runs on Airflow. This is not a criticism — Airflow is a good tool for the job at a certain scale. This guide walks through building production-grade maintenance DAGs with working code, and at each step contrasts the manual approach with the autonomous alternative — so you can see exactly where the architectural boundary lies between workflow orchestration and a purpose-built control plane.

The core tension is structural. Airflow is a workflow orchestrator: it runs tasks on schedules, retries failures, and logs results. Iceberg table maintenance is a control plane problem: it requires continuous health monitoring, event-driven triggers, cross-table coordination, conflict-aware execution, and query-informed layout optimization. The DAG approach works until the gap between these two architectures becomes the dominant source of performance degradation, wasted compute, and engineering toil in your lakehouse.

LakeOps is the autonomous lakehouse control plane referenced throughout — a Rust-based maintenance engine built on Apache DataFusion that connects to your existing Iceberg catalogs (Glue, REST/Polaris, Nessie, S3 Tables) and query engines without moving data or changing pipelines. It replaces cron-scheduled Spark procedures with continuous health monitoring, adaptive triggers that fire when tables actually need maintenance, a native compaction engine that runs 95% faster than Spark at a fraction of the cost, and query-aware layout optimization that sorts data for how you actually query — not just for file size. Understanding the manual DAG approach is still valuable: it clarifies what any maintenance system must do, and the code here works for smaller deployments where the control plane overhead is not yet justified.

What Iceberg maintenance actually does

Every write to an Iceberg table creates a new snapshot, adds manifest entries, and produces data files. Over time, this creates four distinct types of structural debt that degrade query performance and inflate storage cost:

Small files. Streaming pipelines committing every 60 seconds produce thousands of tiny files per day. Each file must be opened, read, and closed by the query engine — turning what should be sequential I/O into random access. A table with 40,000 files where it should have 400 can run 8–12x slower.

Snapshot accumulation. Every commit creates a snapshot. Snapshots pin references to superseded data files, preventing storage reclamation. A table with 5-minute commits accumulates 8,640 snapshots per month — each one adding to the metadata chain that query planners must traverse.

Orphan files. Failed writes, aborted compaction jobs, and concurrent writer conflicts leave data files on storage that no snapshot references. These files are invisible to Iceberg — but fully billable by S3, GCS, or ADLS. On mature lakes, orphans routinely account for 25–40% of billable storage on affected table prefixes.

Manifest fragmentation. Manifests are the index layer between snapshots and data files. Each contains per-file statistics (path, partition values, row count, column min/max) that engines use for predicate pushdown. When manifests fragment — too many small manifests, or manifests that do not align with query patterns — planning time increases because the engine reads more metadata to determine which files are relevant.

Iceberg ships four Spark SQL procedures to address these — one per debt type. The procedures are well-designed. They are not self-running. The question is how to run them: manually via scheduled DAGs, or autonomously via a control plane that monitors table health and fires each operation when the table actually needs it. Both paths use the same underlying operations — the difference is orchestration intelligence. Let us start with the DAG approach, then contrast at each step.

Building the maintenance DAG

Step 1: Snapshot expiration

Snapshot expiration runs first in the pipeline. It releases metadata references to superseded data files, making subsequent orphan cleanup and compaction more efficient (you avoid rewriting files that are about to be garbage-collected).

1from airflow import DAG2from airflow.providers.apache.spark.operators.spark_sql import SparkSqlOperator3from datetime import datetime, timedelta4 5default_args = {6    "owner": "data-platform",7    "retries": 2,8    "retry_delay": timedelta(minutes=5),9}10 11with DAG(12    "iceberg_maintenance_clickstream",13    default_args=default_args,14    schedule_interval="0 2 * * *",  # Daily at 2 AM UTC15    start_date=datetime(2026, 1, 1),16    catchup=False,17    tags=["iceberg", "maintenance"],18) as dag:19 20    expire = SparkSqlOperator(21        task_id="expire_snapshots",22        sql="""23            CALL catalog.system.expire_snapshots(24                table => 'analytics.clickstream',25                older_than => current_timestamp() - INTERVAL 5 DAYS,26                retain_last => 10027            )28        """,29    )

Key parameters:

• older_than — How far back to retain snapshots. 3–7 days for streaming tables; 14–30 days for compliance workloads where time travel is contractually required.
• retain_last — Minimum snapshot count regardless of age. Protects low-write tables from accidentally expiring all snapshots. Set to at least 10; production environments with rollback requirements often use 50–100.
• max_concurrent_deletes — Parallelism for the file deletion phase. On tables with thousands of expired files, increasing this from the default (1) to 10–20 reduces job duration significantly.

The control plane difference. In LakeOps, snapshot expiration is not a scheduled task — it is a continuous health signal. The control plane monitors snapshot depth per table and fires expiration when the count crosses a configured threshold. Retention windows are set once in a hierarchical policy (catalog → namespace → table), not per-DAG. Tables with contractual time-travel requirements get 90-day retention automatically; streaming tables with no compliance needs get 3-day retention. No per-table cron configuration, no stale configs for dropped tables.

Step 2: Orphan file cleanup

With snapshots expired, their file references are released. Orphan cleanup now safely identifies and deletes files that no remaining snapshot references.

1    orphans = SparkSqlOperator(2        task_id="remove_orphans",3        sql="""4            CALL catalog.system.remove_orphan_files(5                table => 'analytics.clickstream',6                older_than => current_timestamp() - INTERVAL 7 DAYS7            )8        """,9    )

Critical safety rule: The retention threshold must be longer than your longest-running write job. A Spark job that runs for 4 hours creates temporary files during execution. If orphan cleanup runs with a 1-hour threshold, it deletes files the job has not committed yet — corrupting the table. The Iceberg default is 3 days; production environments with long-running Flink checkpoints or Spark backfills should use 7 days minimum.

The control plane difference. The 7-day heuristic is conservative by necessity — your DAG cannot know which partitions have active writers. A control plane with partition-level awareness can be more precise: LakeOps tracks which partitions have in-flight jobs, knows the exact duration of active writes, and safely cleans orphans from idle partitions while leaving active ones alone. The safety guarantee is structural — based on real-time writer awareness — not a time-based guess.

Step 3: Data file compaction

Compaction is the most compute-intensive operation. It reads small files, merges them into larger files at a target size, and atomically commits the new file set. Two strategies are available:

1    compact = SparkSqlOperator(2        task_id="compact_files",3        sql="""4            CALL catalog.system.rewrite_data_files(5                table => 'analytics.clickstream',6                strategy => 'binpack',7                where => 'event_date >= current_date() - INTERVAL 7 DAYS',8                options => map(9                    'target-file-size-bytes', '268435456',10                    'min-input-files', '5',11                    'partial-progress.enabled', 'true',12                    'partial-progress.max-commits', '10'13                )14            )15        """,16    )

Strategy choices:

• binpack (default) — Merges files to reach the target size. Does not change row order. Fast, safe, and sufficient for most workloads.
• sort — Rewrites files in a specific column order. Enables statistical pruning: engines check file-level min/max and skip files whose ranges do not match the query filter. Dramatically faster queries, but more compute-intensive to execute.

Production parameters:

• target-file-size-bytes — 256 MB (268435456) is the standard target. Increase to 512 MB for scan-heavy analytics; decrease to 128 MB for latency-sensitive workloads.
• where — Scope compaction to recent partitions. Without this clause, the job rewrites the entire table history — expensive and unnecessary. Always partition-scope your compaction.
• partial-progress.enabled — When true, the job commits intermediate results every max-commits file groups. If the job fails mid-execution, completed groups are preserved. Without this, a failure after 90% completion produces zero output.
• min-input-files — Minimum file count to trigger a rewrite. Set to 5 to avoid rewriting partitions that have only 2–3 files (diminishing returns).

The compaction bottleneck — and the architectural alternative. This is the step where the difference between Airflow-managed Spark and a purpose-built engine is most visible. Each Spark compaction job carries 2–5 minutes of JVM startup, cluster provisioning, and executor allocation before a single byte is read. On tables above 1 TB with position deletes, Spark routinely OOMs — requiring cluster resizing and partition-scoped batching that add more DAG complexity. LakeOps replaces Spark with a Rust-based compaction engine built on Apache DataFusion — no JVM provisioning, no GC pauses, no cluster scaling. Bounded memory means tables that OOM Spark run without special configuration. The engine also analyzes cross-engine query telemetry to determine sort order, enabling data skipping that reduces scan volume by 90%+.

Step 4: Manifest rewriting

Compaction changes the file set. Manifest rewriting consolidates the resulting manifest files — grouping data file entries by partition spec for efficient query planning.

1    manifests = SparkSqlOperator(2        task_id="rewrite_manifests",3        sql="""4            CALL catalog.system.rewrite_manifests(5                table => 'analytics.clickstream'6            )7        """,8    )

Manifest rewriting is lightweight compared to compaction — it operates on metadata, not data files. Run it after every compaction to keep the manifest layer aligned with the current file layout.

The control plane difference. In LakeOps, manifest rewriting is automatically sequenced after every compaction — no separate task configuration needed. The control plane also tracks manifest fragmentation as an independent health signal, triggering rewrites when manifest-to-file ratios degrade even outside of compaction cycles.

Assembling the pipeline

The dependency chain enforces the correct execution order:

1    expire >> orphans >> compact >> manifests

This DAG handles one table. For multiple tables, the common pattern is a parameterized DAG with a loop:

1TABLES = [2    {"name": "analytics.clickstream", "retention_days": 5, "compact_days": 7},3    {"name": "analytics.sessions", "retention_days": 14, "compact_days": 30},4    {"name": "raw.events", "retention_days": 3, "compact_days": 3},5]6 7for table_config in TABLES:8    table = table_config["name"]9    safe_name = table.replace(".", "_")10 11    expire = SparkSqlOperator(12        task_id=f"expire_{safe_name}",13        sql=f"""14            CALL catalog.system.expire_snapshots(15                table => '{table}',16                older_than => current_timestamp() - INTERVAL {table_config['retention_days']} DAYS,17                retain_last => 10018            )19        """,20    )21 22    orphans = SparkSqlOperator(23        task_id=f"orphans_{safe_name}",24        sql=f"""25            CALL catalog.system.remove_orphan_files(26                table => '{table}',27                older_than => current_timestamp() - INTERVAL 7 DAYS28            )29        """,30    )31 32    compact = SparkSqlOperator(33        task_id=f"compact_{safe_name}",34        sql=f"""35            CALL catalog.system.rewrite_data_files(36                table => '{table}',37                strategy => 'binpack',38                where => 'event_date >= current_date() - INTERVAL {table_config['compact_days']} DAYS',39                options => map(40                    'target-file-size-bytes', '268435456',41                    'min-input-files', '5',42                    'partial-progress.enabled', 'true'43                )44            )45        """,46    )47 48    manifest = SparkSqlOperator(49        task_id=f"manifests_{safe_name}",50        sql=f"""51            CALL catalog.system.rewrite_manifests(52                table => '{table}'53            )54        """,55    )56 57    expire >> orphans >> compact >> manifest

This gets you to production. The DAG enforces correct sequencing, handles retries, and supports per-table configuration. For 10–30 tables with predictable batch workloads, it is the right solution.

The configuration scaling problem. This parameterized DAG pattern works at modest scale. At 100+ tables across multiple catalogs, the configuration array becomes a maintenance burden of its own — different retention windows, compaction scopes, target file sizes, sort columns, and retry parameters per table. New tables are added by copying an existing config. Stale configs for dropped tables persist indefinitely. Nobody audits the full configuration because it spans hundreds of lines of YAML.

A policy-based control plane replaces per-table configuration with hierarchical rules: set a policy at the catalog level, override at the namespace for streaming workloads, and add table-level exceptions where needed. Three policies cover 500 tables. Adding a new table requires zero configuration — it inherits the policy from its namespace.

Where the DAG approach breaks

The DAG above works until the lakehouse crosses one of several scaling thresholds. These thresholds are not about Airflow's capacity — they are about the architectural mismatch between workflow orchestration and table management.

Signal 1: Streaming tables degrade between compaction runs

A streaming pipeline committing every 60 seconds to a table with 100 active partitions generates 144,000 new files per day. The nightly compaction DAG at 2 AM merges them. By 10 AM the next day, the table has accumulated 48,000 new small files. Every query between 2 AM and the next 2 AM pays the full scan penalty. The problem is not that compaction failed — it is that the schedule cannot match the write velocity. An event-driven control plane monitors file count continuously and fires compaction when the threshold is crossed — a streaming table might compact every hour, while a batch table compacts once daily after ingestion.

Signal 2: The DAG configuration file is longer than the DAG itself

When per-table configurations include retention windows, compaction scope, target file sizes, sort columns, where clauses, and retry parameters — and when you have 100+ tables across multiple catalogs — the configuration file becomes a maintenance burden of its own. New tables are added by copying an existing config and changing three values. Stale configs for dropped tables persist indefinitely. Nobody audits the full configuration because it spans 600 lines of YAML.

Signal 3: Compaction jobs OOM on large tables

The JVM overhead discussed in the compaction step above becomes the dominant bottleneck at scale. A table with 500 partitions and 200,000 small files overwhelms executor memory. The DAG fix is cluster resizing — larger executors, more memory, higher cost — or partition-scoped compaction with smaller batch sizes, which adds more DAG complexity. Neither addresses the fundamental architecture: each Spark job carries JVM startup, shuffle stages, and GC pauses disproportionate to the actual I/O work. The Rust engine alternative (shown in Step 3) eliminates this entirely — bounded memory, zero-copy Arrow buffers, and no cluster provisioning. See compaction strategies for the full binpack vs sort vs Z-order comparison.

Signal 4: Commit conflicts during maintenance

A compaction job rewriting files in a partition collides with a streaming writer appending to the same partition. Iceberg's optimistic concurrency control detects the conflict at commit time and throws CommitFailedException. With partial-progress.enabled = false (the default), the entire job fails — even if 95% of partitions compacted cleanly. The retry runs the full job again, wasting the compute from the first attempt. With partial progress enabled, the committed groups are preserved but the conflicted partitions remain uncompacted until the next scheduled run. Over time, these are the partitions that accumulate the worst small-file debt — because they are the ones with the highest write concurrency. The exact partitions that need compaction most are the ones hardest to compact with cron-based scheduling.

For deeper analysis of conflict patterns and resolution, see Iceberg commit conflicts: causes, prevention, and recovery.

Signal 5: No one knows which tables are actually healthy

The Airflow dashboard shows green checkmarks: 100% of maintenance DAGs succeeded. But DAG success does not mean table health. A compaction job that rewrites 10 files into 8 files succeeded — but the table still has 4,000 small files. A snapshot expiration that removed 5 snapshots succeeded — but the table still has 8,000 because the retention config is wrong. Without table-level health metrics — small-file ratio, snapshot depth, manifest fragmentation, delete-file count — the team is maintaining tables blind.

When it is time to move

The signals above are not hypothetical. They are the production experience of every team that has scaled Iceberg past 50 tables with mixed workloads. The question is not whether the DAG approach will break — it is when.

The honest engineering answer: if your lakehouse has fewer than 30 tables, all batch, single catalog, single engine, and your team is not on-call for maintenance failures — the DAG approach works. Keep it. The code above is production-grade.

If any of the following are true, the DAG approach is already costing you more than a dedicated control plane would:

• More than 50 tables across multiple catalogs
• Streaming and batch workloads on the same tables
• Multiple query engines (Spark + Trino, or Snowflake + DuckDB)
• Compaction failures that require manual intervention more than once a month
• No dashboard that shows per-table health status
• More than 1 engineer-hour per week spent on maintenance DAG management

Beyond the DAG: query-aware optimization and lake-wide observability

The signals above mark the boundary where DAG-based maintenance stops scaling. Throughout this guide, we have contrasted each manual step with the control plane alternative — adaptive triggers, Rust-based execution, policy-based configuration, and partition-level coordination. Two capabilities remain that have no DAG equivalent at all:

Query-aware layout optimization. Beyond merging small files to a target size, LakeOps analyzes cross-engine query telemetry — which columns appear in WHERE clauses across Trino, Spark, Athena, Snowflake, and DuckDB — and sorts data files around those columns. Statistical pruning skips 90%+ of files before any data is read. Layout simulations let you test sort strategies against real query patterns before committing to a rewrite — projected scan reduction, file skip rate, and query planning impact, all without writing a single file.

Lake-wide observability replaces Airflow task logs with table-level health scoring. Every table continuously classified as Healthy, Warning, or Critical based on structural metrics — small-file ratio, snapshot depth, manifest fragmentation, delete-file count, partition skew. Operations logged with before/after metrics. Insights ranked by severity surface the tables that need attention before queries degrade. The Monitoring view shows operations coverage, readiness state, and optimization timeline across every catalog and namespace — one view that replaces hundreds of Airflow DAG statuses. See lakehouse observability for the full health-scoring model.

Watch demo·LakeOps product demoYouTube ↗

Migrating without disruption

The migration from Airflow maintenance to a control plane is additive, not a cutover. Airflow continues orchestrating your data pipelines — ETL, ingestion, transformation. Only the maintenance responsibility shifts.

1. 1.Connect catalogs — LakeOps discovers every table in your existing Glue, REST, or S3 Tables catalogs. No schema changes, no data movement.
2. 2.Review health — The dashboard surfaces tables you did not know were degraded. Most teams find 15–30% of tables in Warning or Critical state that Airflow DAGs never flagged.
3. 3.Start with one namespace — Enable adaptive maintenance on a single namespace. Observe the results for a week.
4. 4.Retire DAGs — Disable the Airflow maintenance DAGs for namespaces that are now under autonomous management. Keep Airflow for what it does best: pipeline orchestration.

For teams that prefer a gradual approach, LakeOps supports scheduled and manual execution modes alongside adaptive. You can mirror your current Airflow cadence in LakeOps policies, validate the results side-by-side, then switch to event-driven triggers when you are ready.

The code above still works

If you are here for the DAG code, use it. It is the right approach for a small to medium Iceberg deployment with batch workloads and a single engine. Build the maintenance pipeline, enforce the sequencing, scope compaction to recent partitions, enable partial progress, and set up alerting on task failures.

When the lakehouse outgrows it — and you will know because the signals above will be impossible to ignore — the move to an autonomous control plane is not an admission of failure. It is the natural evolution of a data platform that has crossed the threshold from format adoption to operational maturity. The DAGs got you here. The managed lakehouse takes you further.