The AI lakehouse is not a new architecture. It is what happens when your existing data lake gains the intelligence to manage itself — when compaction, maintenance, routing, and observability stop being manual operations and start running autonomously, driven by real signals from the data and the workloads that consume it.
Most teams already have the foundation: Apache Iceberg tables on object storage, one or more query engines, a catalog. What they lack is the operational layer that keeps hundreds or thousands of tables healthy without human intervention — and then extends that health to AI agents as first-class consumers.
This guide covers both dimensions of the AI lakehouse. First, how intelligent automation transforms a passive data lake into a self-managing platform: continuous observability, autonomous maintenance (including query-aware compaction), intelligent routing, and governance policies. Then, how that managed foundation becomes the substrate for agentic AI workloads: MCP interfaces, safety guardrails, and the closed-loop feedback system that lets the lake learn from agent behavior.
LakeOps is the autonomous control plane built specifically for this — a single platform that connects to your existing Iceberg catalogs and query engines (no data movement, no vendor lock-in) and delivers every component of the AI lakehouse: observability, autonomous maintenance and intelligent compaction, multi-engine routing, governance policies, and agentic AI readiness. Teams running LakeOps in production see 80% cost reduction, 12× faster queries, and 95% faster compaction versus Spark — across every connected table, automatically.
What is an AI lakehouse
An AI lakehouse is an open data lakehouse extended with the intelligence to manage itself and the interfaces to serve AI agents. It builds on the foundation you already have — Iceberg tables, open catalogs, multiple query engines — and adds five operational capabilities that turn passive storage into an active, self-optimizing platform.
The five components:
| Component | What it does | Why it matters |
|---|---|---|
| Observability | Continuously classifies every table as Healthy, Warning, or Critical | You cannot fix what you cannot see |
| Autonomous maintenance | Expires snapshots, removes orphans, compacts files (query-aware sort), rewrites manifests, resolves deletes — sequenced correctly, health-driven | Tables degrade silently without it; wrong sequence wastes compute |
| Multi-engine routing | Routes each query to the optimal engine by cost, latency, or throughput | Wrong engine = 10× cost, 5× latency |
| Governance & policies | Declarative rules that inherit across catalogs, namespaces, and tables | Consistency at lake scale without per-table scripts |
| Agentic AI interface | MCP protocol with discovery, analysis, governance tools + safety guardrails | AI agents become first-class consumers |
The first four components are about AI managing your lakehouse — using intelligent automation to keep the platform healthy, fast, and cost-efficient. The fifth is about your lakehouse serving AI agents — exposing governed operations to autonomous consumers through standardized protocols. Both are necessary. But the first four are the prerequisite — you cannot serve reliable AI workloads from degraded tables.
Observability — see your lake's real health
Every AI lakehouse starts with observability. Without continuous visibility into table health, degradation accumulates silently until queries fail or produce wrong results. And you cannot automate maintenance if you cannot detect when it's needed.
Iceberg tables degrade in specific, measurable ways: small files accumulate from streaming, manifests fragment after repeated compaction cycles, snapshots pile up pinning dead storage, delete files slow reads, and partition skew concentrates hotspots. Each form of degradation maps to a specific failure mode — slower planning, higher costs, unreliable results.
The AI lakehouse continuously classifies every table into health tiers: Healthy (structural indicators within bounds), Warning (degradation underway, will reach Critical without action), or Critical (severe fragmentation, queries at risk of timeout or incomplete results). Classifications are computed from real Iceberg signals: file count and size distribution, manifest depth, snapshot accumulation, delete-file ratio, partition skew, and sort-order alignment with actual query patterns.
Beyond classification, the observability layer provides actionable insights — specific diagnoses with severity levels and linked remediation actions. A HIGH-severity insight might report: 'Excessive manifests (92 files, threshold 50, 43% undersized) — severely impacting query planning performance.' Each insight connects directly to the affected table with a specific recommended action.
Cross-engine telemetry completes the picture. The AI lakehouse tracks how every connected engine — Trino, Spark, DuckDB, Athena, Snowflake — uses each table: which columns appear in predicates, query volume, latency distribution, cost per query. This telemetry drives every downstream optimization: compaction sort orders, routing decisions, and maintenance prioritization all depend on knowing how data is actually consumed.
How LakeOps delivers it. The observability platform continuously health-scores every connected table across all catalogs (Glue, Polaris, Nessie, Gravitino, Lakekeeper). Four severity levels (CRITICAL, HIGH, WARNING, LOW) with specific remediation actions. Cross-engine telemetry captures field access frequency across all connected engines. Lake-wide dashboard refreshes automatically. No agents or sidecars to deploy — connects directly to your catalog metadata. See Iceberg Lakehouse Observability for the full setup guide.
Autonomous maintenance — the lake manages itself
The operational model of the AI lakehouse must shift from reactive ("run compaction when someone complains about slow queries") to proactive ("detect degradation signals and remediate before any consumer is affected"). Manual compaction scripts and cron jobs cannot meet this requirement at scale — they either waste compute on healthy tables or miss degraded ones.
Iceberg table maintenance consists of five distinct operations. Each solves a different form of degradation, and the correct execution sequence matters: each operation's output becomes the next one's clean input. Run them out of order and you waste compute or miss reclaimable storage. For the full metadata lifecycle, see Iceberg Metadata Maintenance.
1. Expire snapshots
Every Iceberg write — append, overwrite, delete, compact — creates a new metadata snapshot. Without expiration, snapshots accumulate indefinitely: a table receiving 100 commits/day holds 36,500 snapshots after a year. Each snapshot pins its referenced data files and manifests, preventing garbage collection. A table might show 500 GB of live data while pinning 3 TB of unreachable dead storage. Snapshot expiration removes snapshots older than a retention threshold (e.g. 7 days or 100 snapshots), releasing their pinned files for cleanup. Retention policies must balance cost (shorter = less storage) against time-travel requirements (longer = more reproducibility for AI agents). See Iceberg Retention Policy for strategies.
2. Remove orphan files
Orphan files are data files in object storage that no snapshot references — left behind by failed writes, expired snapshots, or compaction operations that completed in metadata but whose old files were never physically deleted. They consume storage and inflate cloud bills without serving any query. Orphan cleanup scans the storage location, compares against all referenced files in current metadata, and deletes unreferenced files that exceed a safety age threshold (typically 3–7 days, to avoid deleting files from in-progress operations). At production scale, orphan accumulation is significant: one deployment reclaimed 59,831 files totaling 74.8 GB from a single table. See Orphan File Cleanup for the full guide.
3. Compact data files
Streaming ingestion, CDC pipelines, and frequent small appends produce thousands of undersized files. A table with 50,000 × 1 MB files instead of 500 × 100 MB files forces the query engine to deserialize 100× more manifest entries during planning and issue 100× more object storage reads. Compaction merges small files into larger ones (bin-pack) targeting 256–512 MB, or physically reorders them by sort key (sort compaction).
Query-aware sort compaction goes beyond bin-packing to solve the data layout problem. When a query filters WHERE customer_id = 'abc123', the engine checks each file's min/max statistics. If data was written in arrival order, every file's range spans the entire domain — no files can be pruned, and the query reads the entire table. Sort compaction physically reorders data by the columns workloads actually filter on, enabling file pruning that skips irrelevant files entirely. But which columns to sort by? Cross-engine telemetry reveals the answer: capture field access frequency (WHERE, JOIN, GROUP BY) across all connected engines and sort by the most-filtered columns.
Sort strategies include linear sort (single column), Z-order (2–4 columns interleaved for multi-predicate queries), and Hilbert curves (higher-dimensional interleaving). Layout Simulations let you test sort orders on isolated Iceberg branches before applying to production, comparing predicted scan reduction across strategies. See Iceberg Compaction Strategies for the full breakdown.
4. Rewrite manifests
Manifests are the index layer between metadata and data files — each manifest tracks a subset of data files with their partition values, column statistics, and file-level min/max bounds. After repeated compaction cycles, manifests fragment: a 10,000-file table might have 500 manifests with 20 files each instead of 10 manifests with 1,000 files each. Fragmented manifests cause the engine to issue hundreds of S3 GET requests during planning instead of a handful. Manifest rewriting consolidates them into fewer, larger manifests — typically triggered when manifest count exceeds a threshold (e.g. 50) or when average files-per-manifest drops below a minimum. See Iceberg Metadata at Scale.
5. Resolve position deletes
Row-level operations (MERGE, UPDATE, DELETE) in Iceberg V2 produce position delete files that record which rows in which data files should be excluded at read time. The engine must reconcile these at query time — a merge-on-read overhead. As delete files accumulate (common in CDC workloads), query performance degrades: 100 delete files on a partition can add 3–5 seconds per query. Resolving position deletes rewrites the affected data files with the deletions physically applied, eliminating the read-time reconciliation cost. See Delete Files Guide.
The correct sequence
The operations must execute in order: expire snapshots → remove orphan files → compact data files → rewrite manifests → resolve position deletes. Expiring first releases references so orphan cleanup finds the maximum unreferenced files. Compaction then operates on the clean, current dataset. Manifest rewriting consolidates against the final compacted layout. Position delete resolution runs last because compaction may already resolve some deletes as a side effect. Running them independently or out of order wastes compute and misses reclaimable storage. For the complete playbook, see Automating Iceberg Table Maintenance.
Autonomous maintenance must be health-driven, not schedule-driven. A table receiving 1,000 streaming commits per hour needs compaction every few hours. A table loaded once weekly needs compaction once monthly. The system monitors real signals — file count growth, average file size drift, delete-file accumulation, manifest fragmentation — and triggers operations only when structural degradation is detected.
How LakeOps delivers it. Autonomous maintenance runs all five operations in the correct sequence across every connected table, triggered by health signals rather than static schedules. Adaptive Maintenance monitors table activity and fires operations only when degradation is detected. Each operation respects Iceberg's optimistic locking with automatic retry on conflict. The intelligent compaction engine is built in Rust on Apache DataFusion — 95% faster than Spark at approximately 10% of the cost. No JVM, no GC pauses, no OOM failures. Learns sort orders from cross-engine telemetry automatically. Beyond file compaction: resolves position deletes, rewrites manifests, and computes Puffin statistics in one coordinated pass — 51% less data scanned across typical workloads. Full audit trails log every operation with before/after metrics. See Autonomous Iceberg Table Maintenance for the deep dive.
Multi-engine routing — right query, right engine
Most data platforms route all queries to a single engine regardless of query shape. The performance and cost differences are dramatic: a point-lookup on DuckDB returns in 50ms at $0.001; the same query on Trino takes 1.8s at $0.03; on Snowflake, 2.1s at $0.08. An organization running 10,000 queries per day on the wrong engine wastes hundreds of dollars daily and adds hours of unnecessary latency.
The AI lakehouse includes a routing layer that dispatches each query to the optimal engine based on query shape, latency requirements, cost ceilings, and real-time engine health. Clients connect to a single stable SQL endpoint. The routing layer handles engine selection transparently — no client-side changes, no N×M driver configurations.
How LakeOps delivers it. Multi-engine routing dispatches queries across Trino, Spark, DuckDB, Snowflake, Athena, StarRocks, and ClickHouse. Clients connect via PostgreSQL wire protocol, MySQL wire protocol, Arrow Flight, or Trino HTTP. Three routing strategies (cost, latency, throughput) apply per routing group. Automatic failover when an engine degrades. Up to 56% cost reduction with zero client changes. The routing layer is informed by table health — as compaction optimizes layouts, more engines become viable for more query shapes, expanding routing options dynamically. See Multi-Engine Iceberg Architecture for the full design.
Governance & policies — rules at lake scale
At lake scale (hundreds of tables across multiple catalogs), per-table configuration is unsustainable. The AI lakehouse needs declarative policies that encode maintenance rules, retention requirements, and optimization strategies — then inherit them automatically across catalogs, namespaces, and tables.
Governance policies cover: snapshot retention periods (by count and by time), orphan file cleanup thresholds, compaction target file sizes, sort strategies, maintenance schedules, and compliance rules (GDPR deletion, audit retention). Policies can be set at any level — organization, catalog, namespace, or individual table — with lower levels overriding higher ones. When a new table is created, it inherits the namespace's policies immediately without manual configuration.
Retention management is particularly critical for the AI lakehouse. Iceberg's time-travel capability depends on snapshot retention — expire too aggressively and you lose the reproducibility guarantee that AI workloads require. Expire too conservatively and storage costs grow unbounded. The right policy varies by table tier: Gold-layer tables serving AI agents might retain 30 days; Silver-layer transformation tables might retain 7 days; Bronze-layer raw ingestion might retain 3 days. See Iceberg Retention Policy for lifecycle strategies.
How LakeOps delivers it. Declarative policies with inheritance across all connected catalogs (Glue, Polaris, Nessie, Gravitino, Lakekeeper). Set at org/catalog/namespace/table scope. Full version history with rollback. Cross-catalog enforcement — uniform policies regardless of which catalog a table lives in. Coordinated with compaction and maintenance so retention and optimization work together, not against each other. See Iceberg Lakehouse Governance for governance patterns.
Agentic AI — serving AI agents from the managed lake
With the lake self-managing and healthy, the final component makes it accessible to AI agents as first-class consumers. This is not raw SQL access — it is a structured protocol layer that gives agents the same operational intelligence your platform engineers have, through interfaces they can consume programmatically.
The Model Context Protocol (MCP) is the standardized interface between AI agents and external systems. For the AI lakehouse, MCP exposes three categories of tools:
Discovery tools — list_catalogs, list_namespaces, list_tables, get_table_schema. Agents explore the data landscape without pre-configured knowledge. An agent tasked with "find all customer-related tables" discovers them programmatically, inspects schemas, and identifies relevant columns — all through governed, audited tool calls.
Analysis tools — get_table_health, get_insights, get_partition_stats, get_maintenance_history. These expose the operational intelligence that determines whether query results will be reliable. An agent can check table health before querying — if a table is CRITICAL, the agent flags uncertainty in its response or triggers maintenance before proceeding.
Governance tools — trigger_compaction, create_policy, set_maintenance_schedule. Agents can act on the lake, not just read from it. A well-governed agent with appropriate permissions can detect degradation and initiate remediation — the same operational loop your platform engineers execute, but autonomous and continuous. For the full MCP implementation, see MCP for Apache Iceberg.
Safety guardrails are non-negotiable. The AI lakehouse stacks enforcement layers: a ReadOnly guard that blocks DDL/DML for analysis-only agents, a CostEstimate guard that rejects queries predicted to scan beyond a threshold, a PII masking guard that redacts sensitive columns before results reach the agent, and a HumanApproval guard that pauses high-impact operations for manual review. Guards compose per agent, per team, or globally — defense in depth rather than a single permission boundary.
Every tool call is logged with agent identity, timestamp, parameters, and results — a complete audit trail. When an agent produces a surprising business recommendation, you can trace the exact queries, health checks, and snapshot IDs that informed it.
How LakeOps delivers it. The agentic AI platform ships a production MCP server with tools across discovery, analysis, and governance. Connect any MCP-compatible client (Cursor, Claude Desktop, LangChain, custom agents). Per-agent RBAC ensures each agent identity only sees authorized namespaces and tables. Layered guardrails (ReadOnly, CostEstimate, PIIMask, HumanApproval) configurable per agent, team, or globally. Full audit trail feeds into the Events system. Wire compatibility via PostgreSQL, MySQL, Arrow Flight protocols. See the dedicated guide: AI Agents and Iceberg.
The closed loop — a self-optimizing lakehouse
The six components above are not independent — they form a closed feedback loop. Agent queries generate telemetry. Telemetry reveals access patterns. Access patterns drive compaction sort orders. Optimized layouts make more engines viable. More routing options reduce cost and latency. Healthier tables produce more reliable agent responses. More reliable responses increase agent adoption. More adoption generates more telemetry. The loop accelerates.
This is the architectural insight that separates the AI lakehouse from a collection of tools: the system improves itself as it's used. When agents start filtering on columns that no human dashboard uses, the compaction planner detects the new pattern and adjusts sort orders accordingly. When a table's file layout becomes optimal for DuckDB (small, sorted, few files), the routing layer starts dispatching point-lookups there instead of Trino — automatically, without configuration changes.
The closed loop also means table health is not a one-time project. As workloads evolve, the system continuously adapts: new query patterns → new sort orders → new routing decisions → new maintenance priorities. The lake self-optimizes as adoption scales. For the deep dive into how this works at Netflix-like scale, see Intelligent Lakehouse for Everyone.
Building your AI lakehouse — the practical path
The AI lakehouse is not a greenfield project. If you already run Apache Iceberg in production, you're building on top of existing infrastructure. The path is incremental:
Week 1–2: Establish observability. Connect your catalogs and deploy health monitoring across all tables. Classify each as Healthy, Warning, or Critical. Identify the tables with the most severe degradation — these are your highest-priority targets. See Iceberg Lakehouse Observability for the full monitoring setup.
Week 2–4: Deploy autonomous maintenance. Start with Critical tables. Set compaction policies (target file size 256–512 MB), snapshot expiry (hourly for high-write tables, daily for batch), orphan cleanup (daily with 7-day safety threshold), and manifest rewriting (triggered when manifest count exceeds 50). Run the full sequence: expire → orphan → compact → manifest rewrite. Verify file counts drop and table health moves to Healthy. See Iceberg Operational Runbook for step-by-step procedures.
Week 4–6: Enable intelligent compaction and routing. Activate query-aware sort based on cross-engine telemetry. Connect query engines (Trino, DuckDB, Spark, Snowflake). Define routing groups by workload type. Configure cost and latency strategies. Monitor engine selection patterns and cost savings.
Week 6–8: Deploy the agent interface. Connect an MCP server to your AI agent framework. Scope permissions per agent identity. Enable discovery, analysis, and governance tools. Configure guardrails. Test that agents can check table health, query data, and trigger maintenance. Establish audit trails.
Ongoing: Monitor the closed loop. As workloads grow, access patterns shift. Sort orders, routing decisions, and maintenance policies adapt automatically — but review monthly to ensure the system's optimization direction matches business priorities.
The AI lakehouse is operational maturity
The AI lakehouse is not a product you buy or an architecture you design from scratch. It is the operational maturity of your existing data lake — the point where maintenance is autonomous, optimization is query-driven, routing is intelligent, health is continuously monitored, governance is declarative, and agents can safely access governed data through standardized protocols.
Every component serves the same purpose: ensuring that when any consumer — human analyst, BI tool, ML pipeline, or AI agent — queries your data, it gets a fast, correct, auditable answer. The lake takes care of itself so everyone else can focus on their actual work.
For the full managed lakehouse architecture, see the platform overview. For the practical walkthrough of optimization end-to-end, see Iceberg Lakehouse Optimization with LakeOps. For the nine components of a production-ready managed Iceberg lake, see Managed Iceberg in 2026. For the MCP implementation details, see MCP for Apache Iceberg.
