Autonomous Operations
Zero cron jobs, zero scripts
your lake optimizes itself
Connect your catalog. Maintenance, compaction, routing, and governance start running autonomously in under ten minutes.
LakeOpsLast 30 days Optimization Activity
Key Metrics
Storage
-56% reclaimedCPU
-76% reductionRecent Operations
| Operation | Table | Duration | Impact | Time | Status |
|---|---|---|---|---|---|
| Compact Data Files | customer_orders orders | 4s | 1.24 TB, 16 → 1 files | 57 minutes ago | SUCCESS |
| Expire Snapshots | payment_transactions payments | 27s | 8.2 TB | 4 hours ago | SUCCESS |
| Expire Snapshots | inventory_snapshots_20250702 warehouse | 3s | 2.1 TB | 4 hours ago | SUCCESS |
| Rewrite Manifests | raw_clickstream analytics | 1.9s | 3 → 1 manifests | 5 hours ago | SUCCESS |
| Compact Data Files | product_catalog products | 6m 11.3s | 3,008 → 1,256 files | 6 hours ago | SUCCESS |
Lake Events
Open and runs on your stack
The problem
Operational toil that scales
with every table you add
Every new table means another compaction job, cleanup script, routing decision, and monitoring dashboard. At scale, operational burden grows faster than data.
Cron-scheduled compaction
Fixed-interval Spark jobs that fire whether the table needs them or not. Over-run risk, cluster provisioning, and silent failures.
Custom cleanup scripts
Snapshot expiry, orphan removal, manifest rewrite — each in its own script, own schedule, own failure mode.
Manual engine selection
Engineers pick engines per query. Wrong choices waste compute. No one tracks whether queries hit the cheapest path.
Incident-driven operations
A failed compaction at 3 AM cascades into query regressions by 9 AM. Someone gets paged, someone writes a post-mortem.
How it works
Triggered by table health —
not by a schedule
Every table is scored continuously. LakeOps runs only the work that's degraded, in the right order, on the right engine. Cadence and sort strategy adapt as the lake changes.
01 · Table health intelligence
Every table monitored, scored, and prioritized
LakeOps continuously monitors file count, small-file ratio, delete-file depth, snapshot age, manifest bloat, and write velocity across every table in your lake. Each table is classified as Healthy, Warning, or Critical. The most degraded tables get attention first — not the ones that happen to be next in a queue.
- Lake-wide health grid — 786 tables scored in real time
- Priority ordering — worst-degraded tables run first
- Every metric visible in the dashboard — nothing is a black box
| Table | NS | Size | Status |
|---|---|---|---|
| customer_orders | orders | 1.24 TB | HEALTHY |
| payment_transactions | payments | 860 GB | WARNING |
| raw_clickstream | analytics | 4.6 TB | CRITICAL |
| product_catalog | products | 42 GB | HEALTHY |
| user_sessions | analytics | 1.9 TB | WARNING |
| inventory_levels | operations | 320 GB | HEALTHY |
| shipping_events | logistics | 580 GB | HEALTHY |
| search_query_logs | analytics | 3.2 TB | CRITICAL |
02 · Adaptive autonomous maintenance
Every table gets the right operations at the right time
LakeOps sequences snapshot expiry, compaction, orphan cleanup, and manifest rewrite in the correct dependency order — per table, based on its health signals. Cadence adapts automatically: streaming tables run multiple times per hour, batch tables daily, healthy tables are skipped entirely.
- Expire → Compact → Clean → Rewrite — correct dependency order
- Cadence adapts to each table’s write pattern and health score
- No wasted work — expired data is never compacted
Monitoring started at seq #4,812. Everything written from that point on is in scope.
03 · Autonomous query routing
Every query hits the right engine automatically
One SQL endpoint. LakeOps profiles each query’s shape, data volume, and latency target, then routes it to the engine that delivers the fastest result at the lowest cost — Trino, Spark, Snowflake, DuckDB. No connection strings to juggle, no manual engine selection.
- Cost and latency targets per query class — enforced automatically
- Automatic fallback if the primary engine is saturated
- 56% lower query spend across mixed workloads
Engine comparison
Compare engines side-by-side on cost, latency, throughput, and data scanned.
Select engines
Performance comparison
| Metric | Spark | Trino | Athena | Snowflake |
|---|---|---|---|---|
| Query success rate | 99.2% | 99.5% | 99.9% | 99.8% |
| Average runtime | 3.1s | 1.8s | 2.3s | 2.1s |
| Cost per query | $0.04 | $0.03 | $0.05 | $0.08 |
| Total queries | 3,120 | 2,456 | 1,280 | 1,876 |
| Data scanned | 4.2 TB | 2.8 TB | 1.5 TB | 3.5 TB |
Cost vs latency
Lower-left is ideal
Success rate
04 · Declarative policies
Define rules once — enforce everywhere
Retention windows, compaction targets, routing rules, governance constraints, sort strategies — set them as declarative policies. LakeOps applies them across every table, namespace, and engine, and enforces them continuously without manual intervention.
- One policy covers hundreds of tables across multiple namespaces
- Global defaults, namespace overrides, table-level exceptions
- Full audit trail for every policy change and enforcement action
Policies
Manage maintenance, configuration, and lifecycle policies for your data lakehouse
| On | Policy | Type | Next Run | Last Run | Updated | Actions |
|---|---|---|---|---|---|---|
orders_critical | Compact Files | Apr 25, 2026, 8:12 AM | Apr 25, 2026, 03:05 AM | Feb 01, 2025, 3:46 PM | ••• | |
payments_compact | Compact Files | Feb 15, 2026, 12:06 AM | Feb 1, 2025, 02:18 PM | Feb 5, 2025, 4:03 PM | ••• | |
Remove orphan files (e-ip...) For all tables in all catalogs every 7 days | Orphan Files | Apr 25, 2026, 8:12 AM | Apr 25, 2026, 04:07 PM | Jun 23, 2025, 04:01 PM | ••• | |
clickstream_cdc_events_p | Expire Snapshots | Apr 25, 2026, 12:03 AM | Apr 25, 2026, 03:05 AM | Jan 28, 2025, 03:25 PM | ••• | |
sessions_cdc_events_p | Expire Snapshots | Apr 25, 2026, 12:03 AM | Apr 26, 2026, 03:05 AM | Jun 26, 2025, 11:11 PM | ••• | |
global_expire_snapshots Runs snapshot expiration on all tables once a day | Expire Snapshots | Apr 26, 2026, 1:18 PM | Apr 07, 2026, 01:08 PM | Mar 14, 2026, 8:42 AM | ••• | |
manifest_rewrite_weekly Rewrite manifests for all critical tables weekly | Rewrite Manifests | Apr 28, 2026, 2:00 AM | Apr 21, 2026, 02:00 AM | Mar 10, 2026, 9:15 AM | ••• | |
staging_config | Configuration | — | — | Dec 31, 2025, 02:45 PM | ••• |
05 · Self-improving engine
Gets faster and smarter with every run
The Rust compaction engine learns from each execution — optimizing batch sizes, parallelism, and memory allocation. Sort orders adapt to evolving query patterns. Query routing improves as cross-engine telemetry accumulates. The entire system compounds its own performance.
- Compaction: 22 min → 11 min on the same table across consecutive runs
- Throughput climbs from 925 to 1,572 MB/s without any configuration
- Sort keys and routing rules evolve as workload patterns shift
Before — Manual
After — Autonomous
Results
Cut costs and boost performance
Benchmarks from production-grade tables across multiple engines and clouds.
Query speed
After compaction + layout optimization
CPU reduction
Compute hours across all engines
Storage saved
Orphans, snapshots & bloat removed
Table health
Autonomous maintenance keeps every table optimized
Connect in minutes
- no vendor lock-in
Connect catalogs & engines
Only metadata is processed — never retained or stored.
Get visibility & insights
Telemetry reveals table health and actions needed.
Choose your mode
Autopilot, manual approval, or policy-driven.
Lakehouse optimized
Built for enterprise
grade data lakes
SOC 2, SSO, RBAC, dedicated support, and the scale your largest Iceberg lakes demand.
Security & compliance
SOC 2 Type II, encryption, SSO/RBAC, and audit trails for regulated teams.
Scale & control
One control plane for your full lake. Real-time visibility, policies, and predictable performance.
Support & training
Dedicated onboarding, training, and enterprise SLAs. Deploy in VPC or on-prem.
Get started
See LakeOps on your stack
Get a personalized walkthrough with your data and architecture.
Short call, no commitment.
Typically 30 min · Free