In July 2026, Google introduced the Borderless Lakehouse — an architecture built entirely on Apache Iceberg that federates catalogs across AWS, Snowflake, and Databricks, runs BigQuery and Spark on the same tables, and layers autonomous storage optimization underneath everything. A week later, Kyle Weller — Google's Head of Product for the Agentic Lakehouse — published a framework that distills Iceberg performance into six distinct layers, from basic file hygiene to format-aware query planning.
The framework is worth studying. Not because Google's specific products are the only path forward, but because the layered thinking reveals where real performance comes from — and where most lakehouse architectures stop too early.
This guide walks through each layer, explains what Google engineered to address it, and shows how you can build the same capabilities into your own open data lakehouse — regardless of which engines or cloud you run — using an intelligent lakehouse control plane.
The mental model: engines, tables, and the control plane
Before diving into the layers, it helps to establish a mental model that separates three concerns most teams conflate:
- 1.Query engines — how you run SQL. BigQuery, Spark, Trino, DuckDB, Athena, Snowflake, StarRocks. Each optimized for different workload shapes, cost models, and latency profiles.
- 2.Table format + catalog — how data is stored, versioned, and discovered. Apache Iceberg provides the open standard. Catalogs like AWS Glue, Polaris, Nessie, Google's Lakehouse Runtime Catalog, and the Iceberg REST specification provide the metadata layer.
- 3.Control plane — how tables stay healthy and how queries reach the right engine. Compaction, maintenance sequencing, observability, governance, routing, and AI agent enablement. This is the layer most teams either build manually or skip entirely.
Google invested heavily across all three. BigQuery and the Lightning Engine for Spark handle the engine layer. The Lakehouse Runtime Catalog and catalog federation handle the metadata layer. Automatic storage optimization — compaction, garbage collection, clustering — handles the control plane layer.
What makes Google's architecture distinctive is how these layers coordinate. Catalog federation connects remote Iceberg catalogs — AWS Glue, Databricks Unity Catalog, Snowflake Horizon — so BigQuery and Spark can query data across clouds without copying files. Credential vending replaces long-lived storage keys with short-lived, downscoped tokens issued per table path, enforcing least-privilege access at the catalog level. Intelligent cross-cloud caching stores remote data fragments locally after the first read, so subsequent queries avoid repeated egress. Each of these is a control plane function — it sits between engines and storage, managing access, performance, and cost transparently.
LakeOps is an open, engine-neutral control plane that provides this third layer for any Iceberg lakehouse. It connects to your existing catalogs and engines, then autonomously manages compaction, the full maintenance sequence, observability, governance, multi-engine routing, and AI agent access — without replacing any component in your stack.
With that context, here is the six-layer framework.
Layer 1: File and layout hygiene
What it covers: Target file sizes, small-file pressure, basic compaction, and sensible partitioning.
This is where Google starts — and it is table stakes. Every streaming job, every Flink or Spark append, every micro-batch write creates files. Without active management, tables accumulate thousands of undersized files. Each file means another metadata entry, another S3 GET request during query planning, and another unit of work for every engine that touches the table.
Google's Lakehouse for Apache Iceberg includes automatic storage optimization that handles compaction and garbage collection for managed Iceberg tables. BigQuery automatically merges small files and reclaims storage from expired snapshots — no manual triggers, no Spark jobs to schedule. The implementation is specific: files are selected for compaction when their average uncompressed size falls below 50% of a 256 MB target. Compaction triggers automatically after any data mutation, and a forced coalesce runs every 24 hours if eligible data exists — preventing small-file debt from accumulating even during idle periods. Clustering reorders data by the columns you specify, so queries that filter on those columns skip entire files.
Building Layer 1 in your own lakehouse
If you are not on BigQuery managed tables, file hygiene falls entirely on you. The standard approach is Spark-based compaction: schedule jobs, provision clusters, tune JVM settings, handle failures. It works, but it is expensive, slow, and difficult to maintain across hundreds of tables.
LakeOps compaction replaces that entire workflow with an autonomous, purpose-built engine. A Rust-based compaction runtime with Apache DataFusion reads your Iceberg metadata, plans optimal file merges, and writes compacted Parquet directly to your storage. No JVM, no cluster provisioning, no shuffle service — compaction runs continuously at a fraction of the cost and time of Spark-based approaches.
Two strategies are available. Binpack merges small files into optimally-sized ones — fast and effective for most tables. Sort goes further, reordering data within files by query-relevant columns so that predicate pushdown and min/max statistics skip more data at read time. LakeOps selects the right strategy per table based on query telemetry.
The key insight from Google's approach: file hygiene is not a one-time cleanup job. It is a continuous operation that must run in the background, triggered by table activity, not by calendars. A table that receives 10,000 commits per day needs different compaction cadence than one that receives 10.
Layer 2: Metadata efficiency
What it covers: Manifest list and manifest traversal cost, caching, pruning, and statistics quality.
This is the layer most teams overlook. Kyle Weller's framing is precise: many workloads degrade long before data volume itself becomes the bottleneck. A table with 50 TB of well-organized data can query faster than a 5 TB table with 2,000 fragmented manifests, missing column statistics, and thousands of expired snapshots still referenced in the metadata tree.
Streaming workloads make this problem acute. A Flink or Kafka Connect pipeline writing micro-batches every minute produces 1,440 commits per day — each adding a new manifest, a new snapshot, and new metadata files. After a month, the table has 43,000 snapshots and a manifest tree so deep that query planning alone takes longer than the scan. The data is fine; the metadata is the bottleneck.
BigQuery addresses this with the Column Metadata Index (CMETA) — a horizontally scalable index over block and column-level metadata that lets the planner perform fine-grained pruning before scanning any data. CMETA turns metadata that is itself big data (terabytes of stats for petabyte-scale tables) into something the planner can traverse efficiently. It is automatically generated and refreshed at no additional cost.
Google also contributes directly to the Iceberg V4 specification. Engineers from Google, alongside Snowflake, Databricks, Apple, Netflix, and LinkedIn, are shaping adaptive metadata trees — a new metadata structure that flattens the manifest hierarchy, supports single-file commits for streaming workloads, and allows inline changes that background maintenance later rebalances into leaf manifests. The Content Stats proposal, ratified into the V4 spec in May 2026, replaces the old column statistics maps with a typed, structured, columnar representation that engines can process far more efficiently.
Building Layer 2 in your own lakehouse
Outside of BigQuery, metadata efficiency depends on disciplined maintenance. Every Iceberg table needs regular manifest consolidation, snapshot expiration, and statistics computation — operations that most teams either skip or run ad-hoc.
LakeOps maintenance sequences these operations as a coordinated pipeline: expire snapshots → clean orphan files → compact data files → rewrite manifests → compute Puffin statistics. The ordering is deliberate — expiring snapshots first prevents compaction from processing files that are about to be garbage collected. Rewriting manifests after compaction ensures the manifest tree reflects the latest file layout. Computing Puffin statistics (NDV, min/max, null counts) after both steps ensures every engine benefits from accurate column-level stats at planning time.
A table with hundreds of fragmented manifests forces hundreds of S3 GET requests just for query planning. Consolidation reduces this to a handful in a single atomic rewrite. Combined with accurate Puffin statistics, the result is faster planning across every engine that touches the table — Spark, Trino, DuckDB, Athena, Snowflake — without any engine-specific tuning.
Layer 3: Mutation and data-change efficiency
What it covers: Updates, deletes, and merges — position deletes, equality deletes, deletion vectors, residual filtering, and the interaction between mutations and file layout.
This is where, as Weller notes, Iceberg's design has its biggest weaknesses. Every DELETE or UPDATE on an Iceberg table does not modify files in place. Instead, it writes delete files (position deletes or deletion vectors in V3) that engines must reconcile at read time. A table that accumulates thousands of delete files forces every query to perform read-time merge operations, degrading performance proportionally.
The interaction between mutations and file layout compounds the problem. A MERGE INTO on a table sorted by customer_id may touch rows scattered across hundreds of files — each requiring a position delete entry. If the engine cannot avoid rewriting those files entirely, the merge becomes a near-full rewrite. Deletion vectors in Iceberg V3 reduce the per-row overhead (a bitmap vs. a row-level delete file), but the fundamental challenge remains: accumulating mutations create read-time debt that grows with every unapplied change.
Google's BigQuery handles this internally through its storage optimization layer, which automatically resolves delete files during background maintenance — the same compaction process that handles file sizing also applies pending deletes, producing clean output files with zero read-time reconciliation overhead. The Lightning Engine for Spark — built on the open-source Gluten and Velox runtimes — accelerates delete-file reconciliation through vectorized C++ execution, avoiding the JVM overhead that makes these operations particularly expensive in standard Spark.
Building Layer 3 in your own lakehouse
For teams not using BigQuery's managed tables, delete-file debt is one of the most common silent performance killers. Tables look healthy by row count and file size — but query performance degrades steadily as delete files accumulate. Most monitoring tools do not surface delete-file ratios or reconciliation cost.
LakeOps observability classifies table health by multiple signals including delete-file ratios, and surfaces issues at severity levels — CRITICAL, HIGH, WARNING, LOW — so teams can act before queries degrade. The compaction pipeline automatically resolves position deletes during file rewrites: the compacted output contains only live rows, with zero delete files remaining. Every subsequent read avoids the reconciliation overhead entirely.
The practical takeaway: mutation efficiency is not just an engine concern. A control plane that monitors delete-file ratios, triggers compaction when thresholds are crossed, and verifies resolution afterward transforms a per-table, per-engineer burden into an autonomous background process.
Layer 4: Physical layout awareness
What it covers: Sort orders, clustering, column-level stats, and row-group/page skipping.
Layout decisions compound over months of writes. An unsorted table forces every query to scan every file. A table sorted by the wrong columns provides sort-order metadata that engines cannot use for the queries actually running against it. The gap between having column statistics and having useful column statistics is where most lakehouse performance is lost.
Google's BigQuery addresses this through automatic clustering — reordering data based on columns that appear most frequently in query predicates. The History-Based Optimizer (HBO) records actual runtime statistics from prior executions and reapplies proven physical transformations on recurrence of similar query shapes. This is a closed-loop system: the optimizer learns from real workload behavior, not static heuristics.
Building Layer 4 in your own lakehouse
Query-aware physical layout is the single highest-leverage optimization most teams are not doing. Standard compaction merges files by size but ignores query patterns entirely. The result: files are the right size but contain data in an order that does not help any query.
LakeOps sort compaction closes this gap. It observes WHERE, JOIN, and GROUP BY columns across all engines for each table, then sorts data during compaction so that Parquet min/max statistics enable aggressive file and row-group skipping. The sort order is per-table and self-improving — as query patterns shift, the sort columns evolve automatically.
Layout simulations let you test a proposed sort order against historical query patterns before committing. LakeOps runs the simulation on an Iceberg branch, measures the projected impact on data scanned, and shows the result — without touching production data.
The connection to Google's HBO is direct: both systems learn from actual query behavior rather than static configuration. The difference is that HBO operates inside a single engine (BigQuery), while a control plane operates across every engine in your stack. When Trino, Spark, DuckDB, and Athena all query the same table, the sort order should reflect the combined query telemetry from all of them — not just one.
The compound effect between Layer 1 and Layer 4 is worth noting explicitly. Sort compaction is not two separate operations — it is file sizing and layout optimization in a single pass. The output files are both right-sized (Layer 1 hygiene) and sorted by query-relevant columns (Layer 4 awareness). Predicate pushdown works at two levels simultaneously: the engine skips entire files whose min/max ranges do not match the filter, then skips row groups within the remaining files. Neither optimization works well in isolation. Right-sized but unsorted files still require full scans. Sorted but fragmented files still require excessive metadata traversal. The compaction engine that handles both in one pass extracts the full compound benefit.
Layer 5: Execution model
What it covers: The shift from row-at-a-time iterators to vectorized, columnar batch processing — SIMD, cache locality, and native runtimes that bypass JVM overhead.
This is the layer that generates the most headlines. Google's Lightning Engine for Managed Service for Apache Spark compiles Spark physical plans into native C++ instructions using Gluten and Velox, achieving up to 4.9x faster performance with zero code changes. BigQuery's enhanced vectorization autonomously selects SIMD operators, eliminates redundant computation, and operates natively on dictionary and RLE-encoded columns. Databricks has Photon. Onehouse built Quanton on a forked Velox engine.
Google's investments go deeper than just the execution model. Kyle Weller's detailed breakdown of the Lightning Engine reveals re-engineered I/O paths: the engine natively consumes Apache Arrow format within C++ (bypassing the UnsafeRow conversion tax), establishes direct gRPC streams to Google Cloud Storage (skipping the Cloud Frontend), and uses lexicographic listing APIs to fetch all file metadata in a few high-throughput calls at the driver level — replacing the millions of recursive API calls typical of open-source Spark. These are not optimizer improvements; they are storage-boundary optimizations that eliminate overhead before any data processing begins.
The industry is converging on native, vectorized execution. But there is a subtlety that Weller highlights: vectorization alone is not enough. Engine performance matters enormously, but the engine can only work with what the storage layer gives it. A perfectly vectorized scan of a table with 50,000 unsorted, fragmented files is still slow. An engine optimized for SIMD batch processing still spends most of its time on I/O if the files are not organized for the query patterns it serves.
Building Layer 5 in your own lakehouse
The execution model is where the engine layer and the control plane layer intersect most clearly. You choose your engines — and most production lakehouses run more than one. Google runs BigQuery alongside Spark. Most enterprises run Trino for interactive queries, Spark for batch ETL, DuckDB for notebooks and CI/CD, Snowflake or Athena for BI.
The question is: which engine handles which query? The default answer — hardcode connection strings per team or pipeline — leads to suboptimal routing. A point lookup that Trino resolves in milliseconds runs on Spark with 30 seconds of cluster startup overhead. A full-table scan that Spark handles efficiently gets routed to DuckDB and runs out of memory.
LakeOps query routing provides a single SQL endpoint that dispatches each query to the right engine based on workload shape, cost model, and engine health. Routing groups — analytics, etl, reports, bi — each map to a stable endpoint with engine-specific optimization strategies. The routing layer is aware of table health: a compacted, well-sorted table unlocks more engines per query shape, while a fragmented table might be restricted to engines that can handle the overhead.
Layer 6: Format-aware operators and plan specialization
What it covers: Operators and planners that internalize Iceberg's structures — manifests, delete files, clustering, and column stats — so that scans, filters, and joins are specialized rather than treating the table as opaque Parquet.
This is the deepest layer and, according to Weller, where I see some in the community get blinded. A fast engine that treats Iceberg tables as a bag of Parquet files misses the format's most powerful features: manifest-level pruning, partition-spec-aware scan planning, delete-file-aware join ordering, and statistics-driven predicate pushdown.
BigQuery's advanced runtime extends its enhanced vectorization path to open formats, including specialized metadata and scan behavior on Iceberg. Snowflake invested significant engineering to bring Iceberg performance close to native table performance. The format-aware engine understands that a partition spec change does not invalidate existing data, that a position delete file maps to specific row positions in specific data files, and that column statistics in manifests can eliminate entire manifest groups before scanning begins.
Building Layer 6 in your own lakehouse
Format-aware plan specialization is primarily an engine-level concern — you benefit from it by choosing engines that invest in deep Iceberg integration. But the control plane plays a critical supporting role: it ensures the storage layer is in a state that lets format-aware engines extract maximum value.
A format-aware engine that can perform manifest-level pruning benefits enormously from consolidated manifests with accurate statistics. An engine that supports partition-spec-aware planning benefits from partitions that are not skewed or fragmented. An engine that can push down predicates based on column stats benefits from Puffin statistics being computed and current.
The control plane closes the gap between what the format supports and what the engine can use. Without it, even the most sophisticated format-aware engine is working with degraded metadata, bloated manifests, and stale statistics — extracting a fraction of the performance the format was designed to deliver.
This is also where engine choice matters most. Not every engine invests equally in format-aware operators. Evaluate whether your engines can prune at the manifest level (skipping entire manifest groups based on partition specs), handle partition evolution transparently (reading data across old and new partition schemes), and use column statistics from manifests and Puffin files for predicate pushdown. The deeper the engine integrates with Iceberg's metadata structures, the more performance it can extract from well-maintained tables — and the more value a control plane delivers by keeping those structures optimized.
Beyond the six layers: observability and governance
Google's framework focuses on performance optimization. But building like Google also means building the operational infrastructure that sustains performance over time. Tables degrade. Partitions skew. Streaming writes create small-file pressure faster than scheduled compaction can resolve. New engineers join and create tables without sort orders. Delete ratios creep up silently.
An open data lakehouse needs continuous observability — not just monitoring, but actionable intelligence about table health across every catalog and engine.
LakeOps observability provides this through a unified dashboard that classifies every table as Critical, Warning, or Healthy based on multiple health signals: file count and size distribution, manifest depth, snapshot accumulation, delete-file ratios, partition skew, sort-order alignment, and cross-engine query telemetry.
Cross-engine telemetry is particularly valuable. When Trino, Spark, DuckDB, and Athena all query the same table, no single engine's monitoring tells the full story. A table that looks idle in Trino metrics might be a hot path for Spark ETL. A table that seems healthy to Athena might have delete-file ratios that degrade DuckDB reads. The control plane aggregates query patterns, latency, and cost across every engine — revealing the combined access pattern that should drive sort order, compaction priority, and routing decisions.
Governance turns observability into policy. LakeOps policies let you define maintenance rules at the organization, catalog, namespace, or table level — with inheritance, version history, and rollback. A policy that enforces snapshot retention of 7 days across all production tables is defined once and applied everywhere. A namespace-level override for compliance tables extends retention to 90 days. The control plane enforces both, continuously.
The agentic layer: AI agents as lakehouse consumers
Google's vision for the Agentic Data Cloud positions AI agents as first-class consumers of analytical infrastructure. The Data Agent Kit enables agentic workflows that query, wrangle, and troubleshoot data autonomously. BigQuery provides built-in MCP tools that give agents direct access to tables, views, and AI engines.
This is not a future-state vision — it is operational today, and it introduces a new set of requirements for the lakehouse. AI agents generate unpredictable query patterns. They cannot diagnose slow queries caused by table degradation. They cannot wait for a Spark compaction job to finish. When an agent hits a table with thousands of small files, it experiences the full latency penalty and either retries (burning tokens) or returns degraded results.
A control plane purpose-built for agentic workloads needs four capabilities: an agent-native interface (MCP with schema-aware tools), safety guardrails (read-only, cost-estimate, PII-masking, human-approval — stackable per agent, team, or organization), intelligent routing that adapts to agent query patterns, and a self-optimizing storage layer where agent query telemetry feeds back into compaction priorities.
LakeOps provides all four. The closed-loop feedback cycle — agents query → telemetry captured → hot tables prioritized for compaction → routing weights adjusted → next agent query is faster — is the same pattern Google implements inside BigQuery, but available across your entire multi-engine stack.
Consider a concrete example. An analytics agent starts querying an order_events table every few minutes — running parameterized lookups by customer_id and event_date. The control plane detects the repeated pattern across dozens of agent sessions, identifies the table as a hot path, prioritizes it for sort compaction on those two columns, and adjusts the routing weight so subsequent agent queries go to DuckDB (sub-second lookups) instead of Trino (multi-second startup). The agent did not request any of this. It simply issued SQL. The lake improved underneath it — autonomously, in response to observed demand.
Iceberg V4: the spec catches up to the architecture
It is worth noting that the Iceberg specification itself is evolving to support this layered architecture. The V4 spec, actively shaped by engineers from Google, Snowflake, Databricks, Apple, Netflix, and LinkedIn, introduces several changes that directly map to the layers above:
- Relative paths (ratified May 2026) — tables can be relocated across regions or buckets by updating a single catalog pointer, eliminating metadata rewrites. This is Layer 2 efficiency applied to operations.
- Adaptive metadata trees (in design) — a new metadata structure that supports single-file commits for streaming workloads, reducing write amplification. This directly addresses Layer 2 at the format level.
- Content Stats (ratified May 2026) — typed, structured column statistics that replace the old maps, enabling engines to perform Layer 4 and Layer 6 optimizations more efficiently.
- Column families (proposed) — independent storage and evolution of column groups, critical for wide ML feature tables where small updates should not trigger full file rewrites (Layer 3).
A control plane that keeps your tables healthy today will benefit from each of these V4 improvements automatically as engines adopt them. The metadata your control plane maintains — consolidated manifests, accurate statistics, resolved delete files, optimized sort orders — is exactly what V4-aware engines will leverage for deeper optimizations.
Building your own open data lakehouse
Google's architecture works because it is integrated, autonomous, and multi-layered. BigQuery handles Layers 5 and 6. Automatic storage optimization handles Layers 1 through 4. The Lakehouse Runtime Catalog handles metadata. The Borderless Lakehouse handles multi-cloud access. Each component is purpose-built, but they all operate as a coordinated system.
For teams building on open infrastructure — with Trino, Spark, DuckDB, Flink, Athena, or Snowflake as engines, and Glue, Polaris, Nessie, or REST catalogs as metadata stores — the equivalent managed lakehouse architecture requires a control plane that provides the same coordination across a heterogeneous stack.
Here is what that looks like with LakeOps:
Connect — connect your catalogs (Glue, REST, S3 Tables, Polaris, Nessie, Gravitino) and engines. LakeOps discovers every namespace and table, begins collecting metadata telemetry, and gives you immediate visibility into table health across your entire lake.
See — the dashboard classifies every table by health signals. You see which tables have small-file pressure, stale snapshots, missing statistics, high delete-file ratios, or partition skew — before any query slows down.
Choose your autonomy level — start manual, run a compaction on one table, verify the results, then enable policies that automate maintenance across your entire lake. Adaptive maintenance coordinates the full sequence — expire → clean → compact → rewrite — triggered by table activity, not fixed schedules.
Route — connect your engines and define routing groups. The control plane dispatches each query to the optimal engine based on cost, latency, and table health. As tables improve through compaction, more engines become eligible for more query shapes. Read more about multi-engine routing.
Enable agents — expose your lake to AI agents through MCP with layered guardrails. Agent query telemetry feeds back into compaction priorities and routing weights. The lake gets smarter as agents use it. Learn about the MCP interface for Iceberg.
The format is open — the differentiation is in the layers
Google's Kyle Weller said it clearly: The format is open. The real differentiation sits in how many of these layers a system is willing to optimize — especially the deeper ones around mutations, execution model, and format-aware operators.
The companies investing across multiple layers — Google with BigQuery and the Lightning Engine, Snowflake with native Iceberg performance, Onehouse with the Quanton engine — are proving that spec compatibility alone is not enough. The real value comes from the layers above and below the format: the engines that can exploit Iceberg's metadata structures, and the control plane that keeps those structures in the state engines need.
For teams building an open data lakehouse, the takeaway is clear. Choose engines that invest in deep format integration (Layers 5-6). Build — or adopt — a control plane that handles Layers 1-4 autonomously (LakeOps provides this). Add observability and governance so that performance does not degrade over time. Enable multi-engine routing so every query reaches the right engine. And prepare for agentic workloads that will stress every layer simultaneously.
The tools exist. The patterns are proven. Google built them at Google scale. Netflix built them at Netflix scale. You can build the same — without building the infrastructure — today.
