AI agents are supposed to be the next interface for enterprise data. Ask a question in plain language, get a precise answer grounded in your company's numbers. No SQL. No dashboards. No waiting for an analyst to context-switch from three other projects.
The vision is compelling. The reality is that most production AI agents deliver confidently wrong answers — or no answers at all. They hallucinate table names, misinterpret column semantics, generate syntactically correct SQL that returns logically meaningless results, and collapse under the weight of enterprise-scale data they were never designed to navigate.
The failure is not in the models. GPT-4o, Claude, Gemini, Llama — they are remarkably capable reasoners when given the right context. The failure is in what sits between the model and the data: a fragmented, semantically impoverished data stack that gives agents raw access to millions of rows without the structure, versioning, or meaning required to reason correctly.
The core argument is straightforward: Apache Iceberg is not just a storage format — it is the foundation for a live context layer that makes enterprise data AI-ready. Combined with semantic modeling (dbt, metrics layers) and standardized agent interfaces (MCP), Iceberg enables a new class of structured Retrieval-Augmented Generation that grounds agent reasoning in governed, versioned, schema-aware data.
This post expands on that argument. We will cover why agents fail, what the data stack needs to look like to fix it, and how production teams are building AI-ready pipelines on Iceberg today.
Why AI agents fail in production
The conventional explanation for agent failures is that models hallucinate. That is true but misleading. Models hallucinate because they lack sufficient context to do otherwise. The hallucination is a symptom. The disease is context starvation.
Consider what happens when an enterprise deploys a conversational analytics agent. The agent receives a user question — "What was our customer retention rate in EMEA last quarter?" — and needs to translate it into a precise SQL query against the company's data. To do this correctly, the agent must know which tables contain customer data, what "retention" means operationally (is it 90-day cohort retention? Logo retention? Revenue retention?), how EMEA is defined in the data (a region column? a country-to-region mapping table? a filter on currency?), and which quarter boundaries apply (fiscal or calendar?).
Without this context, the agent guesses. It scans available table names, picks one that looks plausible, generates a query against columns that might contain the right data, and returns a number that could be off by an order of magnitude. The user receives a confident, well-formatted, completely wrong answer.
This is not a model problem. Give the same model a clear schema, a metrics definition layer, and sample data, and it will generate the correct query. The problem is that enterprise data stacks were never designed to provide this kind of structured context at inference time.
Silos fragment context. Customer data lives in Salesforce, product usage in Snowflake, financial metrics in a warehouse, behavioral events in a data lake. Each system has its own schema, naming conventions, and access patterns. An agent querying across these systems must reconcile conflicting definitions of the same entity — customer_id in one system is account_id in another, user_id in a third — with no authoritative mapping.
Inconsistent definitions create ambiguity. "Revenue" means something different to finance (recognized revenue per ASC 606), sales (booked ARR), product (MRR from active subscriptions), and marketing (attributed pipeline value). These definitions live in documentation wikis, Slack threads, and the heads of senior analysts. They are not encoded anywhere an agent can discover them at query time.
Tribal knowledge is invisible. Every mature data organization has implicit rules that experienced analysts know but never document. "Don't use the orders table for Q1 2024 — the migration corrupted timestamps." "Always filter out internal test accounts." "The revenue column in the summary table double-counts refunds." An agent with SQL access and no tribal knowledge will trip over every one of these landmines.
Schema drift breaks assumptions. Tables evolve. Columns are renamed, types change, new fields appear, deprecated fields linger. An agent trained on last month's schema generates queries against this month's tables and gets column-not-found errors — or worse, silently queries a column that was repurposed to mean something different.
The cumulative effect is that agents operating on typical enterprise data stacks are unreliable by default. Not because the models are weak, but because the data environment is hostile to automated reasoning.
The bottleneck is the data stack
If the problem is context starvation, the solution is not better prompts or bigger context windows. It is a data stack that produces structured, versioned, semantically rich context as a first-class capability.
Traditional data architectures were designed for human consumers. A dashboard developer knows which table to query, understands the business logic behind a metric, and can validate results against intuition. The architecture only needs to provide storage, compute, and access control. Context lives in the developer's head.
Agent-driven architectures invert this. The consumer has no prior knowledge, no intuition, and no ability to validate results against experience. Every piece of context — schema structure, column semantics, metric definitions, data quality signals, access policies, temporal boundaries — must be explicitly provided at inference time. The architecture must produce context, not just data.
This is where Apache Iceberg becomes foundational. Not as a storage format, but as a live context layer that provides the structural, temporal, and semantic primitives agents need to reason correctly.
Iceberg as the backbone for AI-ready pipelines
Apache Iceberg provides four capabilities that transform lakehouse storage from a passive data repository into an active context layer for AI agents.
Structural context through schema. Iceberg tables carry rich, queryable schemas — column names, types, nullability, documentation strings, and nested structure. Unlike raw Parquet files on S3 where schema information is scattered across individual file footers, Iceberg centralizes schema in table metadata that agents can inspect in a single call. An agent querying an Iceberg table can discover that order_total is a decimal(10,2) with a doc string explaining it represents post-tax, post-discount revenue in USD. This structural metadata eliminates an entire class of agent errors — type mismatches, wrong aggregation functions, misinterpreted units — before any query runs.
Temporal context through snapshots. Iceberg's snapshot-based architecture provides time travel as a first-class operation. Every table maintains a history of immutable snapshots, each representing a complete, consistent view of the data at a point in time. For agents, this means versioned access to training data, reproducible query results, and the ability to answer temporal questions natively. An agent can compare this week's metrics to last week's by querying different snapshots of the same table — no date filters, no WHERE clauses on timestamp columns, no risk of comparing data that was mutated between queries.
Transactional context through ACID. Iceberg provides serializable isolation for concurrent readers and writers. An agent querying a table while a streaming pipeline is appending new data sees a consistent snapshot — not a half-written partition, not a mix of old and new data. This consistency guarantee is non-negotiable for agents that chain multiple queries in a reasoning loop. If the data changes between step 3 and step 7 of a 12-step reasoning chain, the agent's conclusions become logically inconsistent. Iceberg's snapshot isolation eliminates this failure mode entirely.
Evolutionary context through schema evolution. Iceberg's schema evolution capabilities support adding, dropping, renaming, and reordering columns without rewriting data files. More importantly, it maintains a history of schema changes that agents can inspect. An agent encountering a customer_region column can discover that it was renamed from geo_region three months ago and that a legacy_region_code column was deprecated in the same change. This evolutionary history gives agents the context to handle schema drift gracefully rather than failing on unfamiliar columns.
These capabilities are foundational — but they only help if the underlying tables are healthy enough to query efficiently. An agent hitting a table with hundreds of thousands of small files, fragmented manifests, and stale snapshots will timeout before it reaches any structured metadata. LakeOps serves as the operational layer that keeps Iceberg tables AI-ready: autonomous compaction ensures fast scans, manifest consolidation keeps planning times low, and table health classification surfaces degraded tables before agents encounter them. LakeOps connects to existing catalogs — Glue, Polaris, REST, S3 Tables — without moving data, acting as a dedicated control plane that ensures the context layer Iceberg provides is always backed by performant, well-maintained storage.
From ACID to context: time travel for training data versioning
The intersection of Iceberg's time travel and AI training pipelines deserves special attention because it solves one of the most persistent problems in production ML: training-serving skew.
Traditional ML pipelines snapshot training data by dumping it to a timestamped directory — s3://ml-data/training/2026-06-15/. This creates a disconnected copy that drifts from the source table immediately. When the model is retrained, the pipeline reads from a new snapshot, but there is no structural link between the training data and the production table. If the source schema changed, columns were renamed, or data quality degraded between snapshots, the pipeline discovers these problems at training time — hours into a compute-intensive job.
Iceberg snapshots eliminate this pattern. Training pipelines read directly from Iceberg tables at a specific snapshot ID. The snapshot is immutable — it will return exactly the same data regardless of when or how many times it is read. The schema is embedded in the snapshot metadata. The pipeline can validate that the training schema matches the serving schema before reading a single row.
For AI agents, this capability enables reproducible reasoning. When an agent answers a question, it can record the snapshot IDs of every table it queried. A subsequent audit can replay the exact same queries against the exact same data and verify the agent's reasoning chain — something impossible with mutable tables or point-in-time directory snapshots. This reproducibility is a compliance requirement for agents operating in regulated industries (financial services, healthcare, insurance) where decisions must be auditable.
Iceberg's snapshot expiration and retention policies create a natural tension with AI reproducibility requirements. Snapshots consume metadata space and slow query planning as they accumulate. Production tables with streaming ingestion can generate thousands of snapshots per day. The balance between time-travel depth and metadata performance is a table-level configuration decision that must account for agent workloads — agents querying historical snapshots need longer retention than typical analytics consumers.
Structured RAG: using Iceberg metadata for intelligent retrieval
Retrieval-Augmented Generation (RAG) was designed for unstructured data — retrieve relevant text chunks from a vector store, inject them into the LLM context window, generate a response. But enterprise AI workloads increasingly need structured RAG: retrieval that understands table schemas, respects data types, follows relationships, and returns precise numerical results rather than approximate text matches.
Iceberg's metadata layer provides three primitives that make structured RAG dramatically more efficient than naive approaches.
Partition-aware retrieval. Iceberg tables are partitioned by business-relevant dimensions — date, region, customer segment, product category. An agent answering "What were Q2 sales in North America?" does not need to scan the entire sales table. Iceberg's partition metadata tells the retrieval layer exactly which partitions contain relevant data. The agent's query touches only the partitions matching region = 'NA' and date BETWEEN '2026-04-01' AND '2026-06-30' — skipping terabytes of irrelevant data. This is not query optimization in the traditional sense. It is retrieval intelligence — the agent retrieves only the context it needs, reducing both latency and token cost.
Statistics-driven pruning. Iceberg maintains column-level statistics at the file and row-group level — min/max values, null counts, and (via Puffin files) distinct value counts. An agent looking for orders with total > 10000 can skip files where the max value for total is 5000 without reading any data. These statistics function as a pre-retrieval filter that eliminates irrelevant data before it reaches the query engine — the structured equivalent of vector similarity scoring in unstructured RAG, but with deterministic precision instead of probabilistic relevance.
Schema-as-context injection. Instead of stuffing raw table DDL into the LLM prompt (wasting tokens on irrelevant columns), structured RAG uses Iceberg schema metadata to inject only the relevant portions of the schema. An agent answering a revenue question gets the schema for revenue-related columns — names, types, descriptions, partition keys — without seeing the 200 other columns in the table. This targeted context injection improves query accuracy while keeping token costs manageable on tables with wide schemas.
The combination of these three primitives means agents querying Iceberg tables spend less time scanning, use fewer tokens, and produce more accurate results than agents using unstructured RAG or direct SQL generation. The metadata layer acts as a retrieval optimizer that is invisible to the agent — the agent issues a natural-language question, and the infrastructure translates it into the minimal data access required to answer it.
Schema-aware agents: understanding tables, relationships, and evolution
A schema-aware agent does not treat tables as opaque data sources. It understands what columns mean, how tables relate to each other, and how the schema has changed over time. This understanding is the difference between an agent that generates plausible SQL and one that generates correct SQL.
Iceberg provides the foundation for schema awareness through its metadata layer, but production schema-aware agents require three additional capabilities.
Column-level semantics. Iceberg supports documentation strings on columns, but most production tables have sparse or outdated column docs. Schema-aware agents need enriched metadata — business descriptions, data quality annotations, usage examples, valid value ranges — layered on top of the Iceberg schema. Semantic layers (dbt metrics, Atlan business glossaries, Alation catalogs) provide this enrichment. The agent queries the semantic layer to understand that cltv means Customer Lifetime Value, is measured in USD, and is calculated as the sum of all order values for a customer minus refunds and chargebacks.
Relationship discovery. Enterprise data models have relationships — foreign keys, join paths, one-to-many hierarchies — that are rarely encoded in the table format itself. A schema-aware agent needs to discover that orders.customer_id joins to customers.id, that products.category_id joins to categories.id, and that the canonical join path from orders to product categories goes through order_items. Without relationship metadata, agents generate cross-joins, miss required join conditions, or create logically invalid queries that produce inflated result sets.
Evolution tracking. When a column is renamed, an agent trained on the old schema generates queries that fail. Schema-aware agents track Iceberg's schema evolution history and maintain a mapping between old and new names. If geo_region was renamed to customer_region, the agent knows they refer to the same concept and can handle queries using either name. This is particularly important for agents that reference historical documentation, Slack messages, or wiki pages that use deprecated column names.
Together, these capabilities transform agents from SQL generators into data-literate reasoners. The agent does not just know what data exists — it knows what the data means, how it connects, and how it has changed.
The semantic layer gap: from rows to relationships
Iceberg provides the structural and temporal context. But between raw rows and business-meaningful answers lies a semantic gap that Iceberg alone does not bridge.
Consider the question "What is our net revenue retention for enterprise customers?" Answering this requires knowing that net revenue retention is calculated as (starting_mrr + expansion - contraction - churn) / starting_mrr, that "enterprise" means plan_tier = 'enterprise' AND arr > 100000, that MRR is derived from the subscriptions table using monthly_amount for active subscriptions, and that the calculation requires cohort logic comparing this period to the prior period for the same set of customers.
None of this is in the Iceberg table metadata. The schema tells you that a monthly_amount column exists and is a decimal. It does not tell you that this column is an input to the NRR calculation, what the calculation formula is, or what business rules define the customer segments.
This is the semantic layer gap — the space between structured data and business meaning. Closing it requires an explicit metrics and entity definition layer that agents can query alongside the data.
Metrics definitions encode business calculations as version-controlled, queryable objects. A metric definition for net_revenue_retention specifies the formula, the source tables, the join logic, the filter conditions, and the time grain. An agent querying the semantic layer receives the complete calculation — not just a column name that might mean the right thing.
Entity definitions formalize what business concepts mean in terms of data. An entity definition for enterprise_customer specifies the filter conditions (plan_tier = 'enterprise' AND arr > 100000), the source table (customers), and the valid join keys (customer_id). When an agent encounters the term "enterprise customer" in a user question, it resolves it to a precise data filter rather than guessing.
Relationship definitions encode join paths, cardinality, and valid combinations. The semantic layer knows that orders joins to customers through customer_id (many-to-one) and that order_items joins to products through product_id (many-to-one). An agent generating a multi-table query follows the defined join paths rather than inventing its own.
dbt's semantic layer and its MCP server are emerging as the standard approach for bridging this gap. The dbt MCP server exposes metrics, entities, and relationships as tool calls that agents can invoke natively — querying the semantic layer with the same MCP protocol they use to query Iceberg tables. This creates a unified context interface where agents discover both what data exists (Iceberg metadata) and what the data means (semantic layer definitions) through a single protocol.
Production patterns for AI agents on Iceberg
Moving from architecture to implementation, three production patterns have emerged as best practices for deploying AI agents on Iceberg lakehouses.
Governed agent reasoning
Governed reasoning constrains what agents can do while preserving their autonomy within safe boundaries. The pattern has three layers.
First, tool-level governance: the MCP interface exposes only the operations the agent is authorized to perform. A customer support agent gets list_schemas, describe_table, and execute_query with read-only enforcement. It never sees a write or delete tool. The agent's action space is defined by the tools available to it, not by post-hoc filtering of its outputs.
Second, query-level governance: every SQL statement the agent generates passes through a guard chain before execution. Read-only enforcement, row limits, cost estimates, PII masking, and human-approval gates stack per routing group. The agent operates freely within these constraints — it can explore schemas, run aggregations, join tables — but cannot scan petabytes, expose PII, or mutate data. The guardrails are infrastructure, not prompts. They cannot be circumvented by creative prompt engineering.
Third, result-level governance: the data returned to the agent is filtered, masked, or truncated before it enters the LLM context window. Sensitive columns are hashed or excluded. Result sets are capped at a configurable row limit. The agent receives clean, safe, right-sized data — never raw production rows with PII.
Hybrid retrieval
Production agents rarely use pure SQL generation or pure vector search. Hybrid retrieval combines both — using vector similarity to identify relevant tables and columns, then generating SQL to retrieve precise data from those tables.
The flow works as follows. The user's natural-language question is embedded and compared against a vector index of table descriptions, column semantics, and metric definitions. This narrows the search space from hundreds of tables to the three or four most relevant ones. The agent then uses Iceberg schema metadata to understand the structure of these tables and generates SQL that retrieves exactly the data needed to answer the question.
This two-stage approach solves the cold-start problem that plagues pure SQL agents. When an agent encounters a new dataset it has never seen, vector retrieval provides a ranked list of candidate tables without requiring the agent to enumerate every schema in the catalog. Once the candidates are identified, structured retrieval (via Iceberg metadata and the semantic layer) takes over and produces precise results.
Semantic query translation
Semantic query translation converts natural-language questions into SQL by routing through the semantic layer rather than generating SQL directly from the question.
Instead of: user question → LLM → SQL → engine → results, the pattern is: user question → LLM → semantic layer query → resolved SQL → engine → results.
The agent translates the user's question into a semantic query that references metrics and entities by name — "net_revenue_retention for enterprise_customers in Q2 2026". The semantic layer resolves this to concrete SQL using its metric definitions, entity definitions, and relationship definitions. The resolved SQL is then executed against the Iceberg table through the standard query pipeline with full guardrails.
This pattern eliminates the most common source of agent errors: incorrect SQL generation. The agent does not need to know the formula for net revenue retention, the filter conditions for enterprise customers, or the join path between subscriptions and customers. It only needs to know the names of the metrics and entities — which it discovers through the semantic layer's tool interface.
LakeOps MCP: agentic AI on Iceberg in production
The patterns described above require infrastructure that most teams do not have — MCP servers, guardrail pipelines, multi-engine routing, continuous storage optimization, and per-agent observability. Building and operating this infrastructure is a significant engineering investment that diverts resources from the AI applications themselves.
AI agents need data that is structured, versioned, and queryable — exactly what a healthy Iceberg lakehouse provides. LakeOps ensures tables stay AI-ready: well-sorted for efficient retrieval, properly compacted for fast scans, and observable through Agentic AI with MCP.
LakeOps provides the full infrastructure stack for agentic AI on Iceberg as a managed control plane.
Agent-native MCP server. LakeOps exposes list_schemas, describe_table, execute_query, and explain_query as MCP tools that any compatible agent — Claude, LangChain, LlamaIndex, or custom — can connect to with zero integration code. The MCP server handles schema discovery, query routing, guardrail enforcement, and result formatting in a single interface. Agents connect to a stable endpoint URL and inherit the full policy stack automatically.
Composable guardrails. Five guards — ReadOnly, RowLimit, CostEstimate, PIIMask, and HumanApproval — stack per routing group. Every agent query passes through the guard chain before execution. The guards are infrastructure, not application logic — they cannot be bypassed by the agent, the user, or the prompt. Configuration is per-endpoint, not per-agent, so every agent connecting to the same endpoint inherits the same trust boundary.
Self-optimizing storage. The Rust-based compaction engine runs continuously, optimizing tables for the queries agents actually issue. Query-aware compaction analyzes agent access patterns and adjusts sort orders to match — if agents predominantly filter on customer_id and event_timestamp, data is re-sorted on those columns for maximum data skipping. Manifest consolidation, snapshot expiration, and orphan cleanup run as a coordinated pipeline. Tables receiving heavy agent traffic get compacted more frequently — the system monitors query latency per table and automatically elevates compaction priority where agents experience degraded performance.
Per-agent observability. Agent context — agent_id, conversation_id, step_index, tool_call_id — propagates through the entire query pipeline via MCP. Every query is attributed to a specific agent, conversation, and reasoning step. Routing metrics, guardrail audit logs, query shape analysis, and cost attribution are all broken down by agent identity. Session replay reconstructs the full sequence of queries an agent issued during a single interaction — turning query history into an agent reasoning debugger.
The compound effect matters: as storage optimizes for agent patterns, more engines become viable for each query shape. This expands routing options, generates more performance data, and further improves both routing and compaction decisions. The lake gets faster the more agents use it.
Building the AI-ready lakehouse: a practical roadmap
Moving from a traditional Iceberg deployment to an AI-ready lakehouse does not require replacing infrastructure. It requires layering context capabilities on top of what already exists.
Phase 1: Get storage healthy. An agent hitting a table with 200,000 small files will timeout regardless of how sophisticated your MCP layer is. Enable continuous compaction, snapshot expiration, and manifest consolidation. This is a prerequisite, not an optimization — uncompacted tables pay a 5–10x latency penalty that no amount of routing or retrieval intelligence can compensate for. LakeOps connects to existing catalogs (AWS Glue, Polaris, REST catalogs, S3 Tables) and object storage in approximately 10 minutes, with no agents to install and no data to move.
Phase 2: Deploy guardrails. Start with ReadOnly and RowLimit on every agent session. Add CostEstimate with a conservative threshold. PII masking for any table containing user data. The cost of deploying guards too early is zero. The cost of deploying them too late is one bad query that scans your entire lake or exposes customer data in an LLM context window. For storage-level security, a zero-trust approach to S3 access for AI workloads provides an additional layer of defense beneath the application guardrails.
Phase 3: Enrich with semantics. Add column descriptions to Iceberg table metadata. Deploy a semantic layer (dbt metrics, business glossary) that encodes metric definitions, entity definitions, and relationship definitions. Expose these through MCP so agents can discover what the data means, not just what it contains.
Phase 4: Enable structured RAG. Build the hybrid retrieval pipeline — vector search for table/column discovery, structured retrieval for precise data access. Configure semantic query translation so agents route through the metrics layer rather than generating raw SQL. This is where agent accuracy steps up from "sometimes useful" to "production-reliable."
Phase 5: Close the loop with observability. Enable per-agent cost attribution, guardrail audit logging, and session replay. Use the data to identify expensive agents, slow tables, and suboptimal routing decisions. Feed the insights back into compaction priorities, guard thresholds, and routing configuration. The system improves autonomously from this point forward.
The shift from data access to data context
The fundamental shift this architecture represents is from data access to data context. Traditional data platforms answer the question "Can the agent reach the data?" with JDBC connections, API endpoints, and access control lists. AI-ready platforms answer a different question: "Does the agent have enough context to reason correctly about the data?"
Iceberg provides the structural foundation — schemas, snapshots, ACID, evolution. Semantic layers provide the meaning — metrics, entities, relationships. MCP provides the interface — standardized, schema-aware tool calls. Guardrails provide the safety — read-only enforcement, cost limits, PII masking. And storage optimization provides the performance — compacted tables, sorted data, consolidated manifests.
None of these layers is sufficient alone. An agent with MCP access to a poorly compacted table is slow. An agent with fast storage but no semantic layer is inaccurate. An agent with semantics but no guardrails is dangerous. The value is in the composition — a stack where each layer reinforces the others.
This framing resonates because it names the problem precisely: agents are not failing because models are weak. They are failing because we are handing them data instead of context. Iceberg, combined with semantic layers, MCP interfaces, and operational infrastructure like LakeOps, transforms the lakehouse from a data store into a context engine — one that makes every agent interaction grounded, governed, and reproducible.
The organizations that operationalize this stack first will have AI agents that reliably answer business questions, discover insights autonomously, and reason over enterprise data with the same rigor a senior analyst would bring. The rest will have chatbots that hallucinate table names.
Further reading
- Iceberg Lakehouse with AI Agents: A Guide — the five infrastructure components for agentic workloads on Iceberg
- Managed Iceberg Solution — autonomous compaction, routing, and observability as a managed control plane
- LakeOps Documentation — configuration guides for MCP, guardrails, routing, and compaction
- Apache Iceberg Table Health and Maintenance — compaction, manifests, snapshots, and orphan cleanup
- Routing Multiple Query Engines with Apache Iceberg — multi-engine routing with QueryFlux
- LakeOps Agentic AI Solution — MCP connectivity, guardrails, routing, and self-optimizing storage for AI agents
