In 2023, Netflix presented their journey to an Iceberg-only data lake at AWS re:Invent. The architecture they described was not just a migration from Hive to Iceberg — it was a vision for an intelligent lakehouse: a system where tables optimize themselves, metadata stays lean without human intervention, and every query engine connects through a single, well-governed interface.
Netflix built an entire ecosystem to make this work. Polaris for scalable catalog management with REST catalog support. Autotune for background compaction that merges small files automatically. Janitors for orphan file cleanup and snapshot expiration. Metacat for unified metadata access and observability. Each component was purpose-built by a dedicated platform engineering team.
The result was impressive: 25% cost reduction from better compression, compaction, and cleanup. Tables that maintained themselves. Query engines that connected through a single REST endpoint. But the infrastructure behind it — custom services, Kafka event pipelines, CockroachDB-backed metastores, Spark-based maintenance clusters — required the kind of engineering investment that only a handful of companies can sustain.
The question for the rest of the industry became: how do you get the same intelligent lakehouse without building Netflix-scale infrastructure?
LakeOps packages every component Netflix built — and capabilities they did not — into a single control plane that connects to your existing catalogs and storage. This guide walks through the seven components of an intelligent lakehouse and shows how LakeOps provides each one.
What makes a lakehouse intelligent
An intelligent lakehouse is not just a lakehouse with Apache Iceberg tables on S3. It is a system where the gap between writing data and maintaining healthy tables is closed automatically. In a traditional lakehouse, you write data and then manually deal with the consequences — small files, metadata bloat, orphan cleanup, snapshot accumulation. In an intelligent lakehouse, these problems are detected and resolved continuously, without scripts, cron jobs, or 2 AM pages.
Netflix's architecture reveals the components every intelligent lakehouse needs:
Catalog management — a centralized, scalable catalog that decouples metadata operations from query engines and supports the Iceberg REST specification for multi-engine interoperability.
Autonomous compaction — background optimization that monitors table write activity and merges small files into optimally-sized ones without manual triggers.
Metadata lifecycle management — automated snapshot expiration, manifest consolidation, and orphan file cleanup that keeps metadata lean and storage costs bounded.
Observability — unified visibility into table health, file distributions, query performance, and operational history across every catalog and engine.
Policy-driven governance — organization-wide rules for maintenance schedules, configuration standards, and operational guardrails.
Multi-engine support — the ability for Spark, Trino, Flink, Snowflake, Athena, and other engines to share the same tables through standardized interfaces.
Netflix had to build each of these from scratch. LakeOps packages all of them into a single control plane — and adds capabilities Netflix did not share publicly, including layout simulations, multi-engine query routing, and AI agent enablement.
1. Catalog connectivity
What Netflix built
Netflix built Polaris to replace their Hive Metastore with an Iceberg-native catalog backed by CockroachDB. The key innovation was supporting the Iceberg REST catalog specification, which standardized how engines discover and commit to tables — eliminating the need for engine-specific catalog integrations.
How LakeOps provides it
LakeOps takes a different approach to the same problem. Instead of replacing your catalog, it connects to whatever you already run — AWS Glue, DynamoDB-backed catalogs, REST catalogs (including Polaris, Nessie, Lakekeeper, and Gravitino), S3 Tables, or custom implementations. Once connected, LakeOps discovers every namespace and table automatically and begins collecting metadata telemetry.
This matters because most teams are not in a position to build or migrate to a custom metastore. They have hundreds of tables registered in Glue, or they chose a REST catalog early and need to keep it. An intelligent lakehouse should work with your existing catalog infrastructure, not require you to replace it.
Connect multiple catalogs across regions and cloud environments. LakeOps normalizes metadata across all of them into a single view, so you manage one lake — not five disconnected catalogs.
2. Autonomous compaction
What Netflix built
Netflix's Autotune service monitors table write events via SQS and triggers Spark jobs to compact small files in the background. It reacts to table activity, merging hundreds of tiny Parquet files into optimally-sized ones so read performance stays consistent. This is the core of what makes their lakehouse self-maintaining.
How LakeOps provides it
LakeOps provides the same autonomous compaction through a fundamentally different engine. Instead of spinning up Spark clusters for each compaction run, LakeOps uses a purpose-built Rust-based compaction engine with Apache DataFusion that reads Iceberg metadata, plans optimal file merges, and writes compacted Parquet files directly to your storage.
The practical difference is significant. Spark compaction requires provisioning and managing compute clusters, JVM tuning, and handling job failures. The Rust engine runs compaction at a fraction of the cost — benchmarks show 221 seconds versus 1,612 seconds (Spark) versus 6,300 seconds (S3 Tables) on identical 200 GB datasets. That is 95% faster, at roughly $5/TB versus $50/TB for Spark-based approaches. Peak throughput reaches 2,522 MB/s — TB-scale tables compact in minutes, not hours.
The engine is also self-improving. The same 1.192 TB table (balance_snapshots) compacts from 22 minutes on the first run to 18 minutes on the second to 11 minutes on the third — same data, same hardware, zero config changes. The planner learns workload telemetry and improves automatically.
Two compaction strategies are available. Binpack combines small files into larger ones targeting optimal file sizes — fast and effective for most tables. Sort reorders data within files by query-relevant columns, so predicate pushdown and column pruning skip more data. LakeOps analyzes which columns appear in WHERE, JOIN, and GROUP BY clauses and sorts by those columns automatically. On production tables, sort compaction reduces data scanned by 51%.
Each table's compaction can run in one of two modes. Manual: configure settings, save, and click Execute to run once — useful for validating results before automating. Automated: toggle Enabled and set a cron schedule, and LakeOps runs compaction continuously on that cadence. Mix modes freely across your lake — automate compaction on high-ingestion tables while keeping manual control on smaller ones.
3. Metadata lifecycle management
What Netflix built
Netflix built separate janitor services — running continuously — to clean up orphan files and expire old snapshots. Without these services, their exabyte-scale lake would accumulate terabytes of unreferenced data and millions of stale snapshots, degrading both query performance and storage costs.
How LakeOps provides it
LakeOps covers the same ground with four coordinated operations, sequenced as a pipeline so each step's output becomes the next one's clean input.
Snapshot expiration — expire old snapshots beyond a configurable retention period while respecting minimum snapshot counts and concurrent readers. Reclaims metadata and enables downstream storage cleanup. Without expiration, streaming tables accumulate thousands of snapshots per day — each adding metadata overhead to query planning.
Orphan file cleanup — detect and safely remove data files no longer referenced by any snapshot. Targets files from failed writes, aborted jobs, and expired snapshots. Configurable age threshold prevents touching in-progress writes. Production runs have recovered 200 TB of orphan data across 324 tables at near-zero compute cost.
File compaction — merge small data files into optimally-sized ones with query-aware sort orders, as described above.
Manifest optimization — consolidate fragmented manifest files and compute Puffin column statistics (NDV, min/max, null counts) so query planning stays fast as tables grow. A table with 500 manifests forces 500 S3 GET requests just for planning — consolidation reduces this to ~30 in a single atomic rewrite. Every connected engine benefits.
The sequencing matters: expire snapshots first → clean orphans → compact → rewrite manifests. Running these independently as disconnected cron jobs produces conflicts and redundant work — compacting files that are about to be expired, rewriting manifests before compaction changes the file layout. LakeOps coordinates them as a single pipeline where each step feeds the next. A single coordinated run on a 350 TB dataset reduced storage to 230 TB in 10 minutes — $33K/year in combined storage and compute savings from one execution.
Each operation supports the same manual-or-automated pattern: configure, Save, Execute once to verify, then toggle Enabled with a cron schedule for continuous automation. When Adaptive Maintenance is enabled, all four operations are bundled into a single data-driven policy that reacts to table activity — the closest equivalent to Netflix's integrated maintenance approach, without building a single service.
4. Lake-wide observability
What Netflix built
Netflix's Metacat provides a unified API for metadata access across all datasets, abstracting away whether a table is Hive or Iceberg. Combined with Kafka-based event streams, Netflix built observability into every layer of their data platform — but it took years of dedicated infrastructure work.
How LakeOps provides it
LakeOps provides full-stack observability out of the box, without requiring Kafka pipelines or custom event infrastructure.
Dashboard — real-time operations count, query speed gains, cost savings, and resource reduction across every catalog. One view for your entire lake. Connect a catalog and you immediately see: total tables (786), lake size (112.4 PB), query acceleration (12.4x), cost savings ($1.37M), CPU and storage reduction (-76%), and recent operations with before/after metrics.
Table health classification — every table is continuously assessed and classified as Healthy, Warning, or Critical based on file count, size distribution, snapshot accumulation, and metadata health. Critical and Warning tables surface immediately — you know exactly which of your 786 tables needs attention right now.
AI-powered insights — ranked recommendations (Critical and Warning severity) that flag problems like small-file accumulation, excessive snapshots, missing retention policies, and partition skew before they impact queries. Insights are actionable — each one links directly to the optimization that resolves it. Insights resolve automatically as optimizations take effect.
Events audit log — every operation recorded with before/after metrics, duration, and status. "Compact Data Files: customer_orders, 16→1 files, 1.24 TB, 4 seconds." Available per-table or lake-wide, filterable by catalog, type, and status. Full audit trail for compliance and operational review.
Netflix spent years building observability infrastructure across their data platform. With LakeOps, you connect a catalog and immediately see the health of every table, the history of every operation, and AI-generated recommendations for what to fix first. Time to value: minutes, not years.
5. Policy-driven governance
What Netflix built
As Netflix scaled to thousands of Iceberg tables, configuring maintenance per-table became unsustainable. They needed organization-wide rules — policies that applied compaction, retention, and cleanup standards across catalogs automatically.
How LakeOps provides it
LakeOps provides a full policy engine that eliminates per-table configuration at scale.
Maintenance policies automate any single operation (snapshot expiration, orphan cleanup, manifest rewrites, compaction) across a catalog or namespace. Define the operation, set a cron schedule, scope it to the right catalog or namespace, and enable it. Every table in scope inherits the policy automatically — including new tables that are created after the policy is defined.
Adaptive Maintenance bundles all four operations into a single data-driven policy that reacts to table activity. Instead of configuring separate cron schedules for compaction, snapshots, orphans, and manifests, Adaptive Maintenance monitors table state and runs the right operations at the right time. This is the closest equivalent to Netflix's integrated maintenance approach — a single toggle that makes tables self-maintaining.
Configuration policies enforce table settings like Iceberg format version, default file format, and write distribution mode across your organization. Prevent configuration drift before it causes performance degradation.
Each policy is scoped to a catalog, namespace, or specific tables. Per-table policies override broader ones for the same operation — set sensible defaults at the catalog level and override only where needed. Every policy is auditable, versioned, and controllable with one toggle.
This is how you scale from 10 optimized tables to 1,000 without proportionally scaling your platform team. Define the rules once, and every new table that enters a governed catalog inherits the right maintenance configuration automatically.
6. Multi-engine query routing
What Netflix built
Netflix's architecture connects Spark, Trino, Flink, and other engines through the REST catalog endpoint. Each engine reads and writes Iceberg tables through a single interface — but query routing (deciding which engine handles which workload) remains a manual architecture decision.
How LakeOps provides it
LakeOps adds an intelligent routing layer powered by QueryFlux — an open-source, Rust-based SQL proxy with ~0.35ms P50 overhead. Connect Trino, Snowflake, Athena, DuckDB, StarRocks, ClickHouse, and Spark to routing groups, and queries are dispatched to the optimal engine automatically.
Three routing strategies are available — cost-optimized (cheapest engine that meets SLA), latency-optimized (fastest response), and throughput-optimized (highest concurrency). Under the hood, a three-router stack handles the dispatch. The Adaptive router uses pure statistics over query history for repeated templates (0ms decision cost). The LLM router reasons about novel query shapes using a language model with live table statistics. The Semantic router matches by embedding similarity for the long tail. The system converges toward optimal routing without manual rule maintenance.
Routing groups organize engines by workload type. Each group gets its own stable endpoint URL (e.g., storefront-analytics.lakeops.dev), engine pool, query-type scope, priority level, and guardrail stack. Applications and agents connect to one endpoint — routing handles the rest.
| Routing group | Engines | Query types | Priority |
|---|---|---|---|
| Storefront Analytics | Trino, DuckDB | SELECT, AGGREGATE | High |
| Checkout Transactions | Snowflake, StarRocks | INSERT, UPDATE, MERGE | Medium |
| Executive Reporting | Snowflake, Trino | SELECT, JOIN | Medium |
| Catalog ETL | AWS Athena, Spark | INSERT, DELETE | Low |
Production deployments show up to 56% workload cost reduction by routing each query to the cheapest viable engine. DuckDB handles point lookups at 0.5s versus 2.3s on Athena for the same query. Automatic failover reroutes queries when an engine goes down — no client changes, no manual intervention. Per-group concurrency limits and proxy-side queuing protect engines from burst pressure.
For teams running multiple engines, this eliminates the hard-coded engine connections in every application and BI tool. Change your engine fleet — add an engine, swap a backend, adjust routing rules — without changing application code.
7. Layout simulations and AI agent enablement
Beyond Netflix
Netflix's public architecture covers the operational foundation — the components required to keep a lakehouse healthy. LakeOps goes further with two capabilities Netflix has not publicly shared.
Layout simulations
When you change a table's partition scheme, sort order, or compaction strategy, the impact on query performance can be dramatic — in either direction. A sort key that aligns perfectly with one workload might degrade another. Changing a partition scheme rewrites every file in the table. Getting it wrong is expensive.
LakeOps provides layout simulations that run proposed changes on an isolated Iceberg branch of your real data. The system analyzes field access frequency — which columns appear in FILTER, SELECT, and JOIN clauses — across your actual query traffic. You get real metrics: file counts, average sizes, scan reduction, execution times. Run multiple strategies side by side, compare the results, and apply only when you are confident. The branch is discarded after analysis — zero production impact.
This is the safety net that makes autonomous optimization trustworthy. You do not need to guess whether Sort compaction with a specific cluster key will help or hurt — simulate it, review the field access frequency analysis, and decide based on real numbers.
AI agent enablement
Netflix's intelligent lakehouse was built for human analysts and data engineers. The next evolution is enabling AI agents to interact with the same data autonomously.
LakeOps provides an MCP (Model Context Protocol) interface that gives AI agents — Claude, LangChain, LlamaIndex, or custom — safe, structured access to your data lake. Through MCP, agents can discover tables, execute SQL queries with full routing optimization, access column-level statistics without scanning, and trigger maintenance operations — all within configurable guardrails.
Five composable guardrails enforce safety for unsupervised agents. ReadOnlyGuard blocks DDL and DML for agent sessions. RowLimitGuard injects LIMIT N to prevent unbounded result sets. CostEstimateGuard rejects queries exceeding scan thresholds. PIIMaskGuard scrubs sensitive columns before results reach the LLM. HumanApprovalGuard pauses high-stakes operations for manual review. Guards stack per routing group — the endpoint defines the trust boundary.
As AI agent usage grows, LakeOps sees their query telemetry alongside human query telemetry and adjusts table layouts accordingly. Tables that agents query most are compacted first, with sort orders aligned to the predicates agents actually use. Per-agent cost attribution and session replay (reconstructing an agent's full query chain by conversation_id) provide the observability needed to manage agent workloads at scale. The lake self-optimizes as AI adoption grows. For a deep dive, see Iceberg Lakehouse with AI Agents: A Guide.
The compound effect
Each component delivers measurable value on its own. Together, they compound.
Storage costs drop because orphan files are continuously removed, snapshots are bounded, and compaction eliminates small-file proliferation. Production runs have recovered 200 TB of orphan data and reduced 350 TB of storage to 230 TB in a single coordinated execution. Teams report up to 55% storage cost reduction.
Compute costs drop because the Rust compaction engine replaces Spark clusters at 10% the cost ($5/TB versus $50/TB), query-aware sort optimization reduces data scanned 51% per query, and routing sends queries to the cheapest appropriate engine with up to 56% workload cost reduction. Up to 75% CPU cost reduction is achievable.
Query latency improves through compaction (47,000 files → 280), sort optimization (51% less data scanned), manifest consolidation (500 manifests → ~30), and Puffin statistics (aggressive file-level pruning). A 52-second query drops to 5.8 seconds. Across the lake, improvements of up to 12x are documented on production tables.
Engineering time shifts from reactive maintenance — debugging scripts, triaging alerts, firefighting at 2 AM — to proactive platform development. Policies execute continuously, every action is logged and reversible, and the system gets smarter over time. The planner learns workload telemetry. The adaptive router converges toward optimal routing. Sort orders align to real query patterns.
Getting started: ten minutes, not two years
Netflix's intelligent lakehouse took years to build with dedicated teams. LakeOps delivers the same capabilities — and more — in under ten minutes.
LakeOps connects to your existing catalogs and object storage. No agents to install, no data to move, no pipelines to modify. Your data never leaves your account.
1. Connect your catalog — Glue, DynamoDB, REST, S3 Tables, or custom. LakeOps discovers every namespace and table automatically and begins collecting metadata telemetry.
2. Optimize tables — start with manual Execute on a few tables, review results in the Events tab (before/after file counts, duration, data volume), then toggle Enabled with a cron schedule for continuous automation.
3. Scale with policies — define maintenance and configuration policies at the catalog or namespace level. New tables inherit rules automatically. Enable Adaptive Maintenance to bundle all operations into a single data-driven policy.
4. Monitor and act on insights — the Dashboard shows real-time impact. AI-powered Insights flag what needs attention. Events provide full audit history. Table health classification shows exactly which tables are Healthy, Warning, or Critical.
Three guarantees apply: no vendor lock-in (works with your existing catalogs, engines, and storage), no infrastructure changes (connects via metadata — no agents or sidecars), and no data movement (reads metadata and writes optimized files back to your storage).
The intelligent lakehouse is no longer a privilege of companies with 200-person platform teams. The architecture Netflix pioneered — autonomous maintenance, unified observability, policy-driven governance, and multi-engine interoperability — is available to every team running Apache Iceberg in production. LakeOps makes it easy.
Further reading
- Managed Iceberg in 2026: Autonomous Data Lake — the nine components every production data lake needs
- Iceberg Compaction Tools Compared — deep dive on the Rust compaction engine with production benchmarks
- Iceberg Metadata Lifecycle Maintenance — the correct maintenance sequence and why order matters
- 7 Iceberg Lakehouse Cost Reduction Strategies — tactical cost optimization across storage, compute, and maintenance
- Iceberg Lakehouse with AI Agents: A Guide — connecting AI agents to your lakehouse with MCP, guardrails, and routing
