Apache Iceberg
Iceberg Lakehouse Observability: Monitor Table Health, Costs, and Performance
Apache Iceberg does not ship with observability. The seven pillars of lakehouse monitoring and how a control plane makes it operational.
Observability & Governance
See table health
before queries get slow
Small-file ratio, snapshot backlog, manifest count, partition skew, and which engine is scanning what — one dashboard. Set retention and compaction rules once; they apply on every catalog.
76%Cost reduction
12×Faster queries
95%Faster compaction
100%Autonomous
Open and runs on your stack
LakeOpsLast 30 days Optimization Activity
Total Operations
12,211
Last 90 days
Query Speed
12.4×
Avg. acceleration across engines
Cost Savings
$1,374,672
Saved in last 3 months
CPU & Storage
-76%
Last 90 days
Data Optimized
46.8 PB
Last 30 days
Key Metrics
Total Tables
786
Tables in all catalogs
Critical Tables
70
Require immediate attention
Warning Tables
105
Should be addressed or auto-piloted
Healthy Tables
566
Tables in optimal state
Total Data
112.4 PB
Total lake data size
Storage
-56% reclaimed112 TB75 TB38 TB
Storage used
CPU
-76% reduction100%66%33%
Compute hours
Recent Operations
Last 10 operations
| Operation | Table | Duration | Impact | Time | Status |
|---|---|---|---|---|---|
| Compact Data Files | customer_orders orders | 4s | 1.24 TB, 16 → 1 files | 57 minutes ago | SUCCESS |
| Expire Snapshots | payment_transactions payments | 27s | 8.2 TB | 4 hours ago | SUCCESS |
| Expire Snapshots | inventory_snapshots_20250702 warehouse | 3s | 2.1 TB | 4 hours ago | SUCCESS |
| Rewrite Manifests | raw_clickstream analytics | 1.9s | 3 → 1 manifests | 5 hours ago | SUCCESS |
| Compact Data Files | product_catalog products | 6m 11.3s | 3,008 → 1,256 files | 6 hours ago | SUCCESS |
Lake Events
LiveLast 24 hours
Compact Data Files·customer_orders
ecommerce_prod·1.24 TB, 16 → 1 files
4s57 min agoOK
Expire Snapshots·payment_transactions
ecommerce_prod·12 snapshots expired
4.6s1h agoOK
Compact Data Files·raw_clickstream
marketing_events·970 → 87 files
9m 31.6s2h agoOK
Remove Orphan Files·user_sessions
marketing_events·847 MB reclaimed, 1,203 files
1m 12s3h agoOK
Rewrite Manifests·search_query_logs
ecommerce_prod·487 → 12 manifests
2.1s3h agoOK
Expire Snapshots·inventory_levels
warehouse_analytics·62 snapshots, 18.4 GB freed
27s4h agoOK
Compact Data Files·product_catalog
ecommerce_prod·3,008 → 1,256 files
6m 11.3s5h agoOK
Rewrite Manifests·shipping_events
warehouse_analytics·14 → 3 manifests
1.0s6h agoOK
Remove Orphan Files·balance_snapshots
warehouse_analytics·59,831 files, 74.8 GB
13m 6.9s7h agoOK
Compact Data Files·ad_impressions
marketing_events·42,633 → 69 files
2m 18s8h agoOK
In the product
From lake-wide healthto the partition that is skewing
Start with Critical / Warning / Healthy counts. Drill into file histograms, snapshot growth, and which columns queries actually filter on.
See how AI agents use these signals via MCPLake-wide dashboard
Critical, Warning, Healthy — every registered table
After catalogs connect, LakeOps discovers every table and reads Iceberg metadata continuously. The dashboard summarizes health tiers, active insights, and lake scope — so platform teams start triage without writing SQL.
- Critical, Warning, and Healthy counts across all connected catalogs
- Table inventory with records, size, status, and last-modified at a glance
- Refreshes as schemas and namespaces evolve — no custom instrumentation
Table health classification
Score from file size, manifests, snapshots, and deletes
Health is computed from Iceberg signals teams already care about: file count and size distribution, manifest depth, snapshot accumulation, delete-file ratio, partition skew, and sort-order alignment with real query patterns.
- Critical — severe fragmentation or metadata bloat; planning or scans at risk
- Warning — degradation underway; likely to reach Critical without action
- Healthy — structural indicators within bounds you define per environment
Events & audit trail
Every compact, expire, and cleanup — with before/after
Every maintenance operation — compaction, expiration, orphan removal, manifest rewrite — is logged lake-wide and per table with duration, impact, and status. Observability closes the loop: see the problem, act, verify the outcome.
- Filter by catalog, operation type, and success or failure
- Before/after file and manifest counts on every event
- Compliance-ready trail for platform and data-governance teams
Actionable insights
Alerts on small files, manifest sprawl, and snapshot backlog
Insights evaluate tables on a schedule and raise prioritized findings — each tied to a table, severity, and recommended next step. Remediate manually or let policies act on the same signal.
- CRITICAL — partition file explosions and runaway write patterns
- HIGH — manifest counts above threshold, snapshot backlog
- WARNING — partition skew, emerging small-file clusters
- LOW — early drift you can fix before the next compaction window
Table-level insights
Drill into the signals driving each table's health
Beyond lake-wide alerts, every table surfaces its own insights — manifest count vs. threshold, partition skew, small-file accumulation, and more. Each finding links directly to the affected table with severity and recommended action.
- Per-table severity breakdown — from CRITICAL to LOW
- Manifest fragmentation alerts with undersized-file counts
- Partition skew and small-file warnings before they compound
Explore & metrics
File histograms and snapshot growth — no Spark SQL to write
The Explore view and Metrics tab expose the full structural picture: records over time, active files, stale files, delete files, file-size histograms, and snapshot-level growth — the same data you would pull from Iceberg metadata tables, pre-joined and charted.
- Records distribution across recent snapshots — spot write-pattern changes
- File size histogram — % of files in optimal range vs. undersized
- Position and equality delete tracking for merge-on-read tables
Partition drill-down
Which partition has 87 files vs 12 everywhere else
The Partitions view breaks down file counts, byte distribution, and delete-file concentration per partition key. Spot the partitions driving planning timeouts or runaway compaction jobs — and act before they escalate.
- Per-partition file count and byte distribution at a glance
- Delete-file hotspots highlighted across partition keys
- Identify streaming-write explosions in individual partitions
Cross-engine telemetry
Which columns Trino, Spark, and Snowflake actually filter on
LakeOps ingests query telemetry from the engines in your stack. Field-access analysis shows which columns appear in filters and joins; engine-level views show latency and load — so observability informs both triage and downstream optimization.
- SELECT, FILTER, and JOIN frequency per column
- Per-engine query volume and latency trends
- Hot tables and cold tables — prioritize maintenance where it matters
Governance
Compaction, retention, and cleanup
as versioned policies — not tribal scripts
Set defaults at the catalog. Override per table. Every run is logged. New tables inherit the same rules.
Declarative policies
Set rules once — every table follows them
Define compaction targets, snapshot retention, orphan cleanup thresholds, and manifest optimization rules at organization, catalog, namespace, or individual table scope. Policies cascade with inheritance — set defaults at the top, override where needed.
- Catalog-wide or per-table policy scoping with priority levels
- Cron-based scheduling with event-driven override triggers
- Full version history — rollback any policy change in one click
Policies
Manage maintenance, configuration, and lifecycle policies for your data lakehouse
| On | Policy | Type | Next Run | Last Run | Updated | Actions |
|---|---|---|---|---|---|---|
orders_critical | Compact Files | Apr 25, 2026, 8:12 AM | Apr 25, 2026, 03:05 AM | Feb 01, 2025, 3:46 PM | ••• | |
payments_compact | Compact Files | Feb 15, 2026, 12:06 AM | Feb 1, 2025, 02:18 PM | Feb 5, 2025, 4:03 PM | ••• | |
Remove orphan files (e-ip...) For all tables in all catalogs every 7 days | Orphan Files | Apr 25, 2026, 8:12 AM | Apr 25, 2026, 04:07 PM | Jun 23, 2025, 04:01 PM | ••• | |
clickstream_cdc_events_p | Expire Snapshots | Apr 25, 2026, 12:03 AM | Apr 25, 2026, 03:05 AM | Jan 28, 2025, 03:25 PM | ••• | |
sessions_cdc_events_p | Expire Snapshots | Apr 25, 2026, 12:03 AM | Apr 26, 2026, 03:05 AM | Jun 26, 2025, 11:11 PM | ••• | |
global_expire_snapshots Runs snapshot expiration on all tables once a day | Expire Snapshots | Apr 26, 2026, 1:18 PM | Apr 07, 2026, 01:08 PM | Mar 14, 2026, 8:42 AM | ••• | |
manifest_rewrite_weekly Rewrite manifests for all critical tables weekly | Rewrite Manifests | Apr 28, 2026, 2:00 AM | Apr 21, 2026, 02:00 AM | Mar 10, 2026, 9:15 AM | ••• | |
staging_config | Configuration | — | — | Dec 31, 2025, 02:45 PM | ••• |
Retention & compliance
Expire snapshots and physically drop expired data
Set snapshot retention periods, GDPR deletion schedules, and storage tier policies. LakeOps enforces retention rules continuously across every table — no manual scripts, no forgotten cleanup jobs.
- Time-based and count-based retention with safety windows
- GDPR-compliant deletion with full audit trail
- Coordinated with compaction — expired data physically removed
Retention RulesEnforcing
Snapshot retention
Count max: 50
GDPR deletion
Full audit trail
Orphan cleanup
Safety window active
Cross-catalog enforcement
Same rules on Glue, Polaris, Nessie, and Gravitino
Whether you run AWS Glue, Polaris, Nessie, Gravitino, or Lakekeeper — governance policies apply uniformly. No per-catalog scripts, no engine-specific maintenance logic.
- Uniform policy enforcement across AWS Glue, Polaris, Nessie, Gravitino
- Engine-agnostic — works with Trino, Spark, Snowflake, Databricks
- Zero drift between catalog governance and actual table state
Cross-Catalog GovernanceUnified
AWS Glue
1,240 tables
Polaris
860 tables
Nessie
430 tables
Gravitino
215 tables
One policy, all catalogs
Zero drift between governance and table state
Lakehouse Control Plane
Observability is one piece. See the full platform
Health scores and policies sit next to compaction, snapshot expiry, query routing, and MCP access — same catalogs, same tables.
Table Health Overview786 tables
566
Healthy
105
Warning
70
Critical
92%
Optimized
| Table | NS | Size | Status |
|---|---|---|---|
| customer_orders | orders | 1.24 TB | HEALTHY |
| payment_transactions | payments | 860 GB | WARNING |
| raw_clickstream | analytics | 4.6 TB | CRITICAL |
| product_catalog | products | 42 GB | HEALTHY |
| user_sessions | analytics | 1.9 TB | WARNING |
| inventory_levels | operations | 320 GB | HEALTHY |
| shipping_events | logistics | 580 GB | HEALTHY |
| search_query_logs | analytics | 3.2 TB | CRITICAL |
Observability
Learns from query patterns and table signals to decide what to optimize, when, and how — no schedules, no manual tuning.
- Closed-loop sense → plan → execute → learn
- Zero manual scheduling or threshold tuning
- Adapts to workload changes in real time
The gap
Iceberg gives you metadata —
not monitoring or governance
Without continuous observability and policy enforcement, tables degrade silently and governance depends on tribal knowledge instead of automated control.
Iceberg has metadata, not monitoring
Metadata tables expose snapshots, files, and manifests — but no native health scores, alerting, or lake-wide dashboards. Platform teams run one-off Spark SQL until something breaks.
Visibility is split across silos
Object-storage metrics, engine query UIs, and Iceberg catalog APIs each tell part of the story. Correlating a slow dashboard with manifest bloat takes days of manual investigation.
Degradation is silent until queries hurt
Small files, manifest sprawl, and snapshot buildup compound over weeks. Planning time grows, scans widen, and nobody notices until analysts or agents report failures.
Multi-engine lakes multiply blind spots
Trino, Spark, Snowflake, Athena, DuckDB, and Flink each see the same tables differently. Without unified telemetry, you cannot tell which engine or table is actually driving cost and latency.
Resources
Learn more
Apache Iceberg
Data Quality and Table Health: Where Reliability Actually Breaks
Data quality and table health are different failure modes — one breaks business trust, the other breaks performance silently. A practical guide to metrics, monitoring, and automated remediation.
Apache Iceberg
Apache Iceberg Table Health and Maintenance: A Complete Guide
Iceberg tables degrade silently in production. A comprehensive guide to the five maintenance operations, why sequencing matters, and how to automate the full lifecycle.
Connect a catalog.
Health scores show up from Iceberg metadata.
LakeOps reads Iceberg metadata from every connected catalog, classifies table health, and surfaces insights — no agents, no pipeline changes, no manual SQL to get started.
1
Connect catalogs & engines
Only metadata is processed — never retained or stored.
2
Get visibility & insights
Telemetry reveals table health and actions needed.
Table health scores
Optimization opportunities
Cost & performance insights
3
Choose your mode
Autopilot, manual approval, or policy-driven.
Autopilot
Manual
Policies
4
Lakehouse optimized
Queries 10x faster
Cost down 76%
Engines optimized
AIs managed
Tables healthy
Fully governed
No vendor lock-in
No code / infra changes
No data changes
Set up in 10 minutes · Works with your existing stack