If you run a data lakehouse on Apache Iceberg, you are almost certainly maintaining it with Apache Airflow. A DAG that calls Spark SQL procedures — rewrite_data_files, expire_snapshots, remove_orphan_files, rewrite_manifests — scheduled nightly at 2 AM. Maybe a second DAG for the streaming tables that need more frequent compaction. A third for the compliance namespace where snapshot retention is 90 days instead of 7.
This setup works. At 10 tables, it works well. At 50 tables across two catalogs, it starts to strain. At 200 tables with mixed streaming and batch workloads, multiple query engines, and a team that has turned over twice since the original DAGs were written — it becomes the single largest source of on-call pages, wasted compute, and silent performance degradation in your lakehouse. The gap is not operational — it is architectural. Airflow is a workflow orchestrator being asked to be a maintenance control plane.
This guide dissects the five structural pitfalls, explains why each is inherent to the orchestrator model (you cannot fix them with better DAGs), and at each step shows how a purpose-built control plane resolves the problem by design. If your Airflow maintenance still works, this guide tells you what will break and when. If it has already broken, it maps the path forward.
LakeOps is the autonomous lakehouse control plane referenced throughout — a Rust-based engine built on Apache DataFusion that connects to your existing Iceberg catalogs (Glue, REST/Polaris, Nessie, S3 Tables) and query engines without moving data or changing pipelines. What makes it fundamentally different from better DAGs: it monitors every table's structural health continuously, fires maintenance when tables actually need it (not on a cron), sequences operations correctly across the entire lake, avoids conflicts with production writers through partition-level awareness, runs compaction 95% faster than Spark at a fraction of the cost on a native Rust engine, and sorts data files based on cross-engine query telemetry so your queries skip 90%+ of files before reading any data. The five pitfalls below each highlight where this architectural difference matters most.
Why every lakehouse needs active maintenance
Apache Iceberg gives your lakehouse ACID transactions, schema evolution, time travel, and partition evolution — all in a format readable by Spark, Trino, Flink, Snowflake, DuckDB, Athena, and every other major engine. What Iceberg does not give you is a system that keeps tables healthy over time.
Every write creates a new snapshot, adds manifest entries, and produces data files. A streaming pipeline committing every 60 seconds to 100 partitions generates 144,000 new files per day. Each file is typically 1–50 MB — far below the 256 MB–1 GB range where query engines perform efficiently. The manifests that index those files fragment. Snapshots accumulate, pinning references to superseded data. Failed writes and aborted jobs leave orphan files on storage that no snapshot references but S3 still bills for.
Left unmanaged, this entropy is not gradual. A table that performs well in week one can degrade 5–10x in query speed by week eight. The degradation is silent — no error, no alert, just Trino scans that used to take 4 seconds now taking 40, and a storage bill that grows 30% month-over-month with no new data.
The maintenance operations are well-defined. The Apache Iceberg documentation specifies four procedures that address four layers of table debt:
| Operation | What it does | Why it matters |
|---|---|---|
| Snapshot expiration | Removes old snapshots and releases file references | Unbounded snapshots bloat metadata and slow planning |
| Orphan file cleanup | Deletes unreferenced files from storage | Orphans can account for 25–40% of storage cost |
| Data file compaction | Merges small files into optimally sized files | Small files cause full-table scans instead of pruning |
| Manifest rewriting | Consolidates fragmented manifest files | Fragmented manifests slow query planning across all engines |
The operations must run in a specific sequence. Expire snapshots first to release metadata references. Then clean orphans — now safe because expired snapshots no longer reference them. Then compact data files — on a clean state, so you never rewrite files about to be garbage-collected. Then rewrite manifests — on the post-compaction file set, so the metadata reflects the actual layout.
The question is not what to run — the procedures are clear. The question is how to orchestrate them: on a fixed schedule via Airflow DAGs, or continuously via a control plane that monitors table health and fires each operation when the table actually needs it. Airflow is the natural starting choice — it is already in your stack, it handles scheduling and retries, and the Spark SQL procedures are the documented execution path. The combination works until it does not. The five pitfalls below explain why.
Pitfall 1: Fixed schedules ignore table state
An Airflow DAG runs on a cron schedule. The schedule is static — defined when the DAG is written, occasionally updated by hand. The table's actual state is irrelevant to when maintenance runs.
A streaming table that accumulated 50,000 small files in 6 hours waits until the 2 AM cron fires. A batch table that received no new data since last week gets compacted anyway — consuming a Spark cluster for 15 minutes to discover there is nothing to do. A table that crossed the compaction threshold at 10 AM will not be compacted until 2 AM the next day, during which every query against it pays the full scan penalty of fragmented files.
The mismatch compounds with heterogeneous workloads. In any production lakehouse, tables have different write velocities, different file sizes, different retention requirements, and different query patterns. A single schedule — or even a handful of schedule tiers (hourly, daily, weekly) — cannot match the granularity of what each table actually needs. The result is a permanent gap between when maintenance should run and when it does.
LakeOps solves this with Adaptive Maintenance — continuous monitoring of table-level structural signals (file count, average file size, snapshot depth, manifest fragmentation, delete-file ratio) that triggers operations when thresholds are crossed. A streaming table that fragments every hour gets compacted every hour. A batch table that loads once a day gets compacted after ingestion. A table that received no new data is left alone entirely. Every table gets exactly the maintenance it needs, when it needs it — without a human deciding when to schedule it.
Pitfall 2: Per-table DAGs do not scale
At 10 tables, you maintain 10 maintenance DAGs (or one parameterized DAG with 10 configurations). The configurations differ: streaming tables need hourly compaction with aggressive file targets; batch tables need daily compaction with larger targets; compliance tables need 90-day snapshot retention instead of 7.
At 100 tables across 3 catalogs, you maintain 100 configurations with different schedules, different retention windows, different compaction strategies, different where clauses for partition-scoped compaction, and different target file sizes. Every new table requires a new configuration. Every schema change or workload shift may require updating the configuration. The DAGs themselves become a maintenance burden — and the engineer who wrote them is often no longer on the team.
At 500 tables, the Airflow scheduler itself becomes a bottleneck. DAG parsing slows. The metadata database bloats. Concurrent task execution saturates the worker pool. Teams add more Airflow infrastructure to maintain more maintenance DAGs — a recursion that data platform engineers recognize but rarely admit on conference talks.
LakeOps replaces per-table configuration with a hierarchical policy model: set a compaction policy at the catalog level, override at the namespace level for streaming workloads, and add table-level exceptions where needed. Three policies cover 500 tables. Adding a new table requires zero configuration — it inherits the policy from its namespace. Dropping a table removes it automatically. This is the managed lakehouse governance model that scales linearly with organizational complexity, not table count.
Pitfall 3: JVM overhead compounds at scale
Each Spark maintenance job carries JVM startup time (2–5 minutes for cluster provisioning and executor allocation), garbage collection pauses, and per-executor memory overhead. For a single table, this is negligible. Across 100 tables, the aggregate startup time alone is 3–8 hours per maintenance cycle. The compute bill for maintenance can rival the compute bill for production queries.
On tables above 1 TB with complex delete patterns (merge-on-read workloads), Spark compaction routinely OOMs. The fix is cluster resizing — larger executors, more memory, higher cost — or job splitting (compact 10 partitions at a time), which adds more DAG complexity. Tables that should take 4 minutes to compact take 25 minutes because the JVM overhead dominates the actual I/O work.
A Rust-based compaction engine eliminates JVM overhead entirely. LakeOps runs compaction on Apache DataFusion — no executor provisioning, no GC pauses, no cluster scaling. Production benchmarks show 95% faster completion times versus equivalent Spark jobs. Tables that cause Spark to OOM are handled without special configuration. The maintenance compute cost drops from the range of $30–50/TB (Spark clusters) to $3–5/TB (Rust engine). For a 100 TB lakehouse, that is the difference between $3,000/month and $300/month in maintenance compute alone. See compaction at scale for the full benchmark methodology.
Pitfall 4: No cross-operation coordination
Even a well-structured Airflow DAG with expire >> orphans >> compact >> manifests enforces the correct sequence within a single table. It does not coordinate across tables, across catalogs, or with production workloads.
Compaction on Table A may run concurrently with a streaming writer on the same Spark cluster, causing commit conflicts and OCC retries. Orphan cleanup on Table B may fire before snapshot expiration on Table C has finished — and if Table B and Table C share storage prefixes, the cleanup may incorrectly identify in-progress files as orphans. A manifest rewrite that starts at 2:15 AM may collide with a compaction job that is still running from the 2:00 AM schedule.
These coordination failures are intermittent, which makes them hard to diagnose. The compaction job succeeds 28 out of 30 days. On the two days it fails, the retry logic either catches it (masking the root cause) or gives up (leaving the table uncompacted until the next cycle). Over months, intermittent failures accumulate into persistent drift: tables that are nominally maintained but structurally degraded.
A control plane that owns the entire maintenance lifecycle solves this structurally. LakeOps sequences operations per table in the correct order, coordinates across tables to avoid cluster contention, uses partition-level awareness to skip hot partitions with active writers, and implements OCC-aware retry logic that reapplies operations to the specific partitions that conflicted — not the entire table. Maintenance and production writes never collide because the control plane sees both. For deeper context on conflict patterns, see Iceberg commit conflicts: causes, prevention, and recovery.
Pitfall 5: Observability is an afterthought
Airflow provides DAG-level observability: task status, duration, retry count, and logs. It does not provide table-level observability: how many small files does each table have? Which tables are degraded? How many orphan files are accumulating? What is the query-time impact of the current file layout?
Without table-level health metrics, maintenance runs blind. The DAG succeeded — but did it actually improve the table? A compaction job that rewrites 10 files into 8 files succeeded by Airflow's definition, but it did not solve the small-file problem. A snapshot expiration that removed 5 snapshots succeeded, but the table still has 4,000 snapshots because the retention window is misconfigured.
LakeOps closes this gap with continuous health scoring: every table classified as Healthy, Warning, or Critical based on structural metrics — small-file ratio, snapshot depth, manifest fragmentation, delete-file accumulation, and partition skew. Tracked per table, trended over time, and surfaced as actionable insights ranked by severity. When a table degrades, the control plane triggers maintenance immediately. When a table is already healthy, it does not waste compute running unnecessary operations. This closed-loop model — observe health, trigger maintenance, measure impact, adapt — is what makes the difference between maintenance as a chore and maintenance as infrastructure.
The five pitfalls are structural
These pitfalls are not bugs in your DAGs. They are structural limitations of using a workflow orchestrator for a control plane problem.
Airflow was designed to orchestrate data pipelines — extract, transform, load — with static dependency graphs and time-based schedules. Table maintenance requires dynamic, state-driven orchestration: monitoring table health continuously, triggering operations based on structural thresholds, coordinating across tables and engines, adapting to workload changes in real time, and providing table-level observability that feeds back into the next decision.
You can add health checks to your DAGs. You can parameterize schedules. You can build custom operators that inspect table metadata before deciding whether to compact. Teams do this — and they end up with 2,000 lines of Python glue code, a custom metadata store, a health-scoring system, a conflict-avoidance layer, and an alerting pipeline. At that point, you have built a control plane inside Airflow. It is harder to maintain than the original DAGs, harder to debug, and harder to hand off to the next engineer.
What the control plane adds beyond fixing pitfalls
You have seen the individual pieces throughout — adaptive triggers for Pitfall 1, hierarchical policies for Pitfall 2, the Rust engine for Pitfall 3, partition-level coordination for Pitfall 4, and health scoring for Pitfall 5. Two additional capabilities have no DAG equivalent at all:
Query-aware layout optimization. Beyond merging small files to a target size, LakeOps analyzes cross-engine query telemetry — which columns appear in WHERE clauses across Trino, Spark, Athena, Snowflake, and DuckDB — and sorts data files around those columns. Statistical pruning skips 90%+ of files before any data is read. Layout simulations let you test sort strategies against real query patterns before committing to a rewrite — projected scan reduction, file skip rate, and query planning impact, all without writing a single file.
Lake-wide operations monitoring. Beyond per-table health, the Monitoring view shows operations coverage, readiness state, and optimization timeline across every catalog and namespace. You see what ran, what is pending, and what needs attention — in a single view that replaces checking hundreds of Airflow DAG statuses.
Migration path: Airflow to autonomous
Replacing Airflow maintenance does not require replacing Airflow. Most teams run LakeOps alongside their existing orchestration — Airflow continues to schedule ETL, ingestion, and transformation pipelines. Maintenance shifts to the control plane.
The migration is incremental:
- 1.Connect catalogs. Point LakeOps at your existing Iceberg catalogs (Glue, REST, S3 Tables). Table discovery is automatic — every table in the catalog is visible within minutes.
- 2.Review health. The dashboard shows which tables are degraded and why. Most teams discover tables they did not know were unhealthy — the silent degradation that Airflow DAGs cannot surface.
- 3.Enable adaptive maintenance. Start with a single namespace or catalog. Adaptive Maintenance bundles compaction, snapshot expiry, manifest rewrites, and orphan cleanup into one policy that triggers on health signals. See the LakeOps quick start for the full walkthrough.
- 4.Retire DAGs gradually. As each namespace moves to autonomous maintenance, disable the corresponding Airflow DAGs. Keep Airflow for pipeline orchestration — it is excellent at that. Remove the maintenance burden it was never designed for.
For teams that prefer more control, LakeOps also supports scheduled and manual modes — cron-based execution, fixed intervals, or one-off runs. You can start with scheduled policies that mirror your current Airflow cadence, validate the results, then switch to adaptive when you are comfortable.
The bottom line
Airflow is a great workflow orchestrator. It is not a lakehouse maintenance system. The five pitfalls — fixed schedules, per-table DAGs, JVM overhead, missing coordination, and blind-spot observability — are not operational failures. They are architectural mismatches between a tool designed for pipeline orchestration and a problem that requires continuous, state-driven, conflict-aware, query-informed table management.
At 10 tables, the mismatch is invisible. At 50 tables, it is annoying. At 200 tables with mixed workloads across multiple engines, it is the dominant source of performance degradation, wasted compute, and engineering toil in your lakehouse.
The alternative is not more Airflow. It is a dedicated control plane for your data lakehouse — one that observes every table continuously, triggers maintenance when the table needs it, sequences operations correctly, avoids conflicts with production writes, sorts data for how you actually query, and gives you lake-wide observability that Airflow's task logs never will.
Your Airflow DAGs got you this far. The lakehouse outgrew them. That is not a failure — it is a signal that the maintenance problem has matured into a systems problem, and systems problems need purpose-built solutions.
