AI Agents for Apache Iceberg Maintenance and Compaction

Every production Iceberg lakehouse shares the same uncomfortable truth: tables degrade silently, continuously, and in ways that compound. Small files accumulate from streaming ingestion. Snapshots pile up and bloat metadata trees. Orphan files from aborted writes inflate storage bills. Manifests fragment until query planning takes longer than the scan itself. Left unattended, a healthy table becomes a performance liability in weeks.

The traditional response — cron-scheduled Spark jobs, hand-tuned Airflow DAGs, and team-specific runbooks — works at small scale. At 50 tables, an engineer can reason about each one. At 500 tables across multiple catalogs and engines, manual maintenance becomes the bottleneck: the same team debugging compaction failures at 2 AM is the team that should be building data products.

This is where AI agents change the equation. Not as a cosmetic layer on top of existing tooling, but as autonomous systems that reason about table health, decide what maintenance each table needs, execute operations in the correct sequence, and learn from the outcomes. This article is a deep technical guide to how AI agents automate Iceberg table maintenance — from the mechanics of each operation to the reasoning loop that makes it intelligent.

The Maintenance Burden: What Iceberg Tables Actually Need

Apache Iceberg provides the primitives for table maintenance, but none of them run themselves. Every production table requires a continuous cycle of four core operations:

Compaction merges small data files into optimally-sized files (typically 256–512 MB). Without it, streaming tables accumulate thousands of tiny files per partition. A single Kafka-ingesting table can generate 3,000+ files per day at 4 MB each; within a week, an engine planning a query must open and read statistics from over 20,000 files before executing a single scan.

Snapshot expiration removes old snapshots beyond a retention window. Every write creates a new snapshot, retaining references to data files that cannot be garbage-collected. A table receiving 3.2 snapshots per hour accumulates over 2,300 per month — and the metadata tree grows so deep that planning alone takes longer than the scan.

Orphan file cleanup removes data objects on S3, GCS, or ADLS that no live snapshot references. In a typical production lake, orphans can account for hundreds of terabytes of dead storage — one real production scan found approximately 200 TB of dead data across 324 tables, roughly 1.8 million orphan files costing $4,000 per month.

Manifest rewriting consolidates fragmented manifest files — Iceberg's index layer that maps data files to snapshots. A table might have 487 manifests where 12 would suffice, and every query plan must open every one.

These operations are interdependent. Running them independently on separate schedules leads to wasted work: compacting files that are about to be expired, cleaning orphans before expiration releases newly unreferenced files, or rewriting manifests that compaction will immediately invalidate. The correct sequence matters — and most manual setups get it wrong.

The Manual Maintenance Nightmare

Today, most teams are running Spark compaction jobs on a cron schedule. Compact nightly at 2 AM. Expire snapshots at 3 AM. Run orphan cleanup on weekends. Some tables get compacted too often, wasting compute. Others not enough, silently degrading performance. Engineers are manually tuning file sizes, expiry policies, and cleanup schedules — table by table, DAG by DAG.

This approach fails in three fundamental ways:

Over-maintenance wastes compute. A nightly compaction job runs on every table regardless of write activity. A streaming table genuinely needs hourly compaction. A reference table updated quarterly needs none. Fixed schedules treat both identically. At $50/TB on Spark, compacting a 2 TB table that doesn't need it burns $100 for zero benefit.

Under-maintenance degrades performance. Consider a concrete scenario: a high-velocity CDC table generates 10,000 small files (averaging 3.2 MB each) over 48 hours between nightly compaction windows. An analyst runs SELECT * FROM clickstream WHERE event_date = '2026-09-14' AND customer_id = 12345 — the engine must open all 10,000 files, read statistics from each manifest entry, and scan far more data than necessary. What should be a 2-second query takes 24 seconds. Dashboards time out, analysts file tickets, and the SLA is blown — all before the next compaction run.

No awareness of table context. A cron job cannot decide that a table should use sort-order compaction because 89% of queries filter on customer_id. It cannot reason that snapshot expiration should run before compaction because 2,928 stale snapshots hold references to files that should be garbage-collected. It cannot prioritize a critical table over a healthy one.

The AI Agent Approach: Health-Driven, Autonomous Maintenance

AI agents represent a fundamentally different model. Instead of fixed operations on fixed schedules, agents reason about what each table needs based on real-time signals, then decide what to do, when to do it, and how to sequence operations safely.

An agent does not compact a table because it is Tuesday at 2 AM. It compacts because the table's small-file ratio crossed 42%, delete files are amplifying read cost, and detected query patterns indicate a sort-order compaction on customer_id and event_date would yield a 12× improvement in scan efficiency.

This requires three capabilities that traditional scheduling lacks:

1. 1.Continuous observability — real-time health scoring across every table, from file layout to metadata structure to query patterns.
2. 2.Contextual reasoning — interpreting health signals in the context of each table's workload, engine mix, and business criticality.
3. 3.Coordinated execution — running the right operations in dependency order, with conflict awareness, retry logic, and verification.

Tools like LakeOps provide the infrastructure that makes this possible: an autonomous control plane that scores table health, monitors every table continuously, and runs a purpose-built Rust compaction engine at 95% faster and 90% cheaper than Spark. LakeOps already automates maintenance without any AI agent involvement. AI agents add the reasoning layer on top: they can explain what happened, predict what's coming, and make nuanced decisions about strategy. Through MCP (Model Context Protocol), agents get the eyes, the reasoning context, and the hands to maintain tables autonomously.

Deep Dive: Compaction Strategies — Bin-Pack vs Sort vs Z-Order

Compaction is the highest-impact maintenance operation, and the one where intelligent strategy selection matters most.

Bin-Pack: Fast File Consolidation

Bin-pack merges small files into target-sized files without reordering rows — essentially file-level concatenation. It is the fastest and cheapest strategy.

When to use it: High-frequency streaming ingest where files arrive at 1–50 MB. Also right for ML training pipelines and batch ETL where queries always full-scan, or tables with natural write ordering.

What it does not do: Improve data skipping. Min/max statistics reflect the random order data arrived, not how it gets queried. Bin-pack reduces file-count overhead but doesn't make individual queries faster through better layout.

What bin-pack is not enough for: Tables with dominant query filter patterns. If 89% of queries on a table filter on customer_id, bin-pack will merge files to the right size — but every file will still span the entire customer_id range, so the engine cannot skip any of them. You need sort compaction to unlock data pruning.

Concrete impact: A streaming raw_sdk_events table: 42,633 small files (averaging 0.2 MB) → bin-packed to 69 files at 512 MB target → query planning drops from 18 seconds to 0.4 seconds. File-open overhead drops 99.8%. S3 GET request costs drop proportionally — fewer files means fewer API calls on every query.

An agent reasons: "This table receives 3,000 files per day from Kafka, average size 4.2 MB. No dominant query pattern detected. Bin-pack to 512 MB, run every 2 hours."

Sort-Order: Align Data with Query Patterns

Sort compaction globally sorts all records by one or more columns, writing files where each covers a narrow, non-overlapping value range. Parquet min/max statistics become highly selective — engines skip 90%+ of files.

When to use it: Tables where 70%+ of queries filter on the same 1–2 columns. Time-series filtered by timestamp, transactional data filtered by customer_id, event tables filtered by type. Higher write amplification than bin-pack — daily or post-batch cadence is typical.

Concrete impact: A 4.6 TB customer_orders table where 89% of queries filter on customer_id and 76% on event_date. Before: 970 files in append order, queries scan 4.6 TB. After sort on (customer_id, event_date): 87 files at 256 MB, properly sorted. A query for WHERE customer_id = 12345 AND event_date BETWEEN '2026-01-01' AND '2026-03-31' skips 92% of files — scanning ~460 GB instead of 4.6 TB. Latency drops from 12.1s to 1.5s. Compute cost drops 76%.

An agent reasons: "89% of queries filter on customer_id, 76% on event_date. Sort compaction would reduce scanned data from 4.6 TB to ~460 GB. Run daily after batch load completes."

Z-Order: Multi-Dimensional Clustering

Z-order applies a space-filling curve to cluster data across 2–4 columns simultaneously. Unlike linear sort (which strongly optimizes the first column and weakly optimizes subsequent ones), Z-order distributes clustering benefit evenly.

When to use it: Ad-hoc analytical tables where queries filter on different column combinations unpredictably. Most expensive strategy — run weekly or during off-peak windows. Clustering quality degrades above 4 columns.

Concrete impact: An ad_impressions table queried by three teams with different filters. After Z-order on (campaign_id, event_date, geo_region): ~46% file skip rate across all patterns. Hilbert curves (available via LakeOps's Rust engine) push this to ~57%.

An agent reasons: "Unpredictable filter combinations across campaign_id, event_date, and geo_region in roughly equal frequency. Z-order across all three during the weekly off-peak window."

How to Choose: A Decision Framework

The decision tree is straightforward once you have the signals:

1. 1.No dominant query pattern, or full-scan workloads → Bin-pack. Cheapest, fastest, reduces file count without rewriting data layout.
2. 2.70%+ of queries filter on the same 1–2 columns → Sort. The scan reduction from data pruning far outweighs the higher rewrite cost.
3. 3.Multiple columns queried in unpredictable combinations → Z-order on the top 2–4 filter columns. Balanced clustering across dimensions.
4. 4.Table is idle (no recent writes) → Skip entirely. An agent saves the compute a cron job would waste.

The critical insight is that strategy selection depends on measurable properties that change over time: query patterns shift when new dashboards launch, write patterns change when pipelines are refactored, table size grows as data accumulates. An AI agent continuously re-evaluates these signals and adapts — something no static configuration can do. LakeOps observes actual query patterns across all connected engines (Trino, Spark, Snowflake, Athena, DuckDB) and selects the optimal sort order per table automatically. When patterns change, the sort order adapts. For a deep breakdown, see the Iceberg compaction strategies guide.

The Rust/DataFusion Engine: Why It Changes the Economics

Traditional compaction runs on Spark — a general-purpose distributed framework. Compaction is fundamentally I/O-bound, but Spark runs it with JVM overhead, GC pauses, executor provisioning, shuffle stages, and idle cluster costs. This is massive overkill.

LakeOps's compaction engine replaces Spark with purpose-built Rust powered by Apache DataFusion:

• Zero-copy Arrow memory — no serialization between stages
• Bounded memory with disk spill — no GC pauses, no OOM kills. A 1.2 TB table that OOMs Spark finishes without special configuration
• Native Parquet I/O — direct column reads/writes without Java interop
• Lock-free parallelism — single-process execution, no executor coordination
• Self-improving planner — same table: 22 min → 11 min across consecutive runs, throughput climbing from 925 to 1,572 MB/s, zero config changes

Production benchmarks (200 GB, 600M rows, same hardware):

Across 5.5 TB and 10 production tables: 101K → 19K files (81% reduction), 2,522 MB/s peak throughput, 551M deleted rows cleaned. Compaction at 10% the cost of Spark.

This cost shift is transformative. At $50/TB, sort-compacting a 5 TB table costs $250 — you think twice. At $5/TB, it costs $25 — you run it as often as the table needs. Operations that were previously too expensive become routine. Aggressive, continuous compaction becomes economically viable across every table.

The Full Maintenance Lifecycle

Snapshot Expiration

An agent balances time travel requirements (regulatory: 30 days; analytics: 3–7 days), metadata performance, and downstream dependencies. Rather than a global retention window, it tailors per table. Production impact: a single run on a table with 23,183 snapshots removed 2,928 snapshots and 5,819 files, reclaiming 263 MB of manifest data in 4 minutes. On another table, 22,034 snapshots and 675,510 files expired, reclaiming 179 GB.

Orphan File Cleanup

AI agents enforce age thresholds (only files unreferenced 3–7 days are candidates), run cleanup strictly after expiration, and verify no active readers are affected. A lake-wide sweep cleaned approximately 200 TB of dead data across 324 tables in under 30 minutes. See the small files and storage bloat guide for details.

Manifest Rewriting and Delete Resolution

Agents trigger manifest rewriting only after compaction finalizes the file set. A real example: search_query_logs with 487 manifests → 12 in 2.1 seconds. For merge-on-read tables, LakeOps physically applies position and equality deletes during compaction — eliminating delete files and the MoR tax in one pass, with Iceberg V3 deletion vectors supported natively.

How Agents Reason: The MCP Intelligence Layer

LakeOps scores every table as Healthy, Warning, or Critical and exposes these scores via MCP, giving agents a real-time view of the entire lake. With 27 tools spanning discovery, analysis, and governance, an agent can:

• Call get_table_health for health score and maintenance status
• Call get_maintenance_signals for compaction/expiration/rewrite needs with projected trigger times
• Call analyze_table_maintenance for a full decision workflow with actionable recommendations
• Call analyze_critical_triage to rank critical tables for on-call response

Any MCP-compatible agent — Claude, LangChain, Cursor, or custom builds — connects with zero integration code via standard Postgres, MySQL, or Arrow Flight protocols. Layered guardrails (ReadOnly, CostEstimate, PIIMask, HumanApproval) ensure safe operation. See the full agentic AI solution.

Agents track write patterns to classify tables and set maintenance cadence automatically:

• Streaming tables (Kafka, Flink) produce thousands of small files per hour. They need frequent bin-pack compaction — every 1–2 hours — with periodic sort compaction daily on settled partitions.
• Batch tables loaded once per day compact immediately after the load completes, with sort or Z-order applied.
• CDC tables with merge-on-read accumulate position and equality delete files that amplify read cost. They need aggressive compaction with low delete-file thresholds (3–5 files).
• Idle tables with no recent writes are skipped entirely, saving compute that a cron job would waste.

Cross-engine query telemetry determines optimal sort orders. When Trino, Spark, Snowflake, and DuckDB all read the same table, their query patterns may diverge. An agent aggregates access patterns, weights filter columns by frequency and selectivity, simulates candidate sort orders, and picks the layout that maximizes data skipping for the dominant pattern. When new dashboards or agents change access patterns, the sort order adapts automatically.

LakeOps: The Autonomous Maintenance Control Plane

The LakeOps platform provides every component the AI agent pattern requires:

Continuous telemetry from Iceberg metadata, object storage, and query engines. Health-driven triggers that replace fixed schedules — maintenance fires when a table needs it, not when a clock says so. The Rust engine executing compaction at 2,522 MB/s peak throughput, 95% faster and 90% cheaper than Spark. Declarative policies that enforce compaction thresholds, retention windows, and sort strategies across every catalog continuously — new tables inherit them automatically. Layout simulations that test sort orders on Iceberg branches before touching production. Full event logging with before/after metrics for audit, compliance, and SOC 2.

The Autonomous Loop: Monitor → Analyze → Plan → Execute → Verify

Monitor: Agents observe table state through MCP — file counts, sizes, small-file ratios, snapshot accumulation, manifest counts. Tables are scored automatically.

Analyze: For degraded tables, the agent determines why. The observability layer surfaces insights at four severity levels — CRITICAL, HIGH, WARNING, LOW — with specific recommendations. A signal like POOR_FILE_DISTRIBUTION (940 small files under 8 MB) alongside EXCESSIVE_SNAPSHOTS (847 snapshots pinning 1.2 TB) tells the agent the root cause is unchecked streaming ingestion, not a configuration error.

Plan: Dependency sequencing (expiration → compaction → cleanup → manifest rewrite), strategy selection (bin-pack/sort/Z-order), parameter tuning (file sizes, delete thresholds, sort columns from cross-engine telemetry), and priority scheduling (critical first).

Execute: Atomic commits via Iceberg OCC, hot-partition avoidance, partial progress, automatic retry, full event logging.

Verify: Did file counts converge toward target? Did query planning time improve? Did the sort order actually reduce data scanned? If a sort order is not improving query times, the planner adjusts. If a table continues to degrade despite compaction, the agent escalates to investigate root cause — perhaps the write pattern changed, or the partition scheme needs evolution.

This verify step is what makes the loop learning. Each execution refines future decisions. Compaction performance improves across runs as the engine learns workload patterns. Sort orders adapt as query patterns evolve. Scheduling intervals tighten or relax based on observed ingestion rates and degradation velocity. For the full architecture, see Autonomous Iceberg Table Maintenance.

Before and After: What This Looks Like in Practice

Before: A platform with 324 tables across 3 catalogs. Compaction runs nightly for the 20 "most important" tables. 304 tables have no maintenance. Over 6 months: 200 TB of orphan files ($4,000/month S3 costs), raw_clickstream with 42,633 small files (queries 8× slower), search_query_logs with 487 manifests (+2.1s planning overhead per query). Three engineers spend ~15 hours/week on maintenance scripts.

After (LakeOps + AI agents): All 324 tables monitored continuously. Health scores flag degradation before it reaches queries. raw_clickstream is compacted from 42,633 → 69 files in 2 minutes 18 seconds on the Rust engine — what would take Spark over an hour finishes in under 3 minutes. search_query_logs manifests are rewritten from 487 → 12 in 2.1 seconds. 200 TB of orphan files are cleaned in under 30 minutes. Queries run 12× faster across the lake. Storage costs drop 56%. Compute costs drop 76%. The three engineers reallocate 15 hours per week to building data products instead of debugging cron jobs.

From Reactive Firefighting to Autonomous Operations

The difference is structural. Manual approaches are reactive — you discover degradation after users complain. Cron-based approaches are blind — they run regardless of need. AI agents invert this: they detect degradation before it reaches queries, prioritize automatically, adapt to changing workloads, and learn from every execution.

• No more 2 AM compaction firefighting. The engine handles conflicts, retries, and logs everything.
• No more manually triaging tables. Health scores identify critical tables automatically.
• No more growing Airflow DAG libraries. Policies define intent; the control plane executes.
• No more silent 40,000-file accumulations. Continuous monitoring catches degradation in real time.

Tables stay healthy as a baseline. Queries run 12× faster. Storage costs drop 56%. Compute costs drop 76%. Engineering time shifts from reactive maintenance to building data products.

The Iceberg table format gives you the primitives for table maintenance — compaction, snapshot expiration, orphan cleanup, manifest rewriting. But primitives alone do not keep a lake healthy. You need intelligence: the ability to observe every table's state, reason about what each one needs, execute the right operations in the right order, and learn from outcomes.

That intelligence can come from manual scripts and human judgment — which works at small scale and breaks at production scale. Or it can come from an autonomous control plane with AI agents that reason about your lake continuously. The metadata lifecycle and maintenance optimization guide covers the full operational picture, from snapshot trees through storage reclamation.

Ready to automate your Iceberg table maintenance? LakeOps connects to your existing catalogs in minutes — no agents to deploy, no data movement, no pipeline changes. Tables are scored, maintained, and optimized automatically. Start free at lakeops.dev.