Quick Start
Connect your Iceberg catalog and have autonomous maintenance running in under 10 minutes. No agents to install, no data movement, no pipeline changes.
1. Connect your catalog
Go to Data > Catalogs > + Add Catalog. Pick your catalog type, enter connection details, and LakeOps discovers every namespace and table automatically. Health analysis begins immediately.
Connect multiple catalogs across regions and cloud environments.
2. Review your lake health
Once connected, LakeOps reads Iceberg metadata and surfaces what needs attention — before you change anything. No agents, no SQL to write.
- •Dashboard — first screen after login. Health breakdown (Critical / Warning / Healthy), storage trends, operations count, and storage savings across every catalog.
- •Tables (Data > Tables) — every table scored as Healthy, Warning, or Critical from small-file ratio, snapshot backlog, manifest count, delete-file accumulation, and partition skew. Filter by status to find the worst first.
- •Insights (Data > Insights) — findings ranked by severity: poor file distribution, excessive snapshots, excessive manifests, partition skew, and missing retention policies.
- •Maintenance tab (per-table) — debt scores, accumulation rates, projected trigger times, and partition-level compaction analysis.
This is a good time to explore a few tables and understand what LakeOps found before enabling automation.
3. Optimize your tables
LakeOps gives you two ways to run maintenance. Pick the approach that fits your team — you can switch per table or fleet-wide at any time.
Adaptive Maintenance
Bundles compaction, snapshot expiry, manifest rewrites, and delete-file handling into a single data-driven policy. No cron schedules — operations trigger automatically when table signals cross thresholds.
- •Per-table — Optimization tab > toggle Adaptive on
- •Fleet-wide — Manage > Policies > + Create > Adaptive Maintenance
Scheduled & Manual
Configure each operation independently with cron schedules, fixed intervals, or manual-only execution. Run any operation once via Execute, or set it on a cadence and let it run continuously.
- •Per-table — Optimization tab > configure & schedule each operation
- •Fleet-wide — Manage > Policies > per-operation policies with cron
Many teams start with scheduled or manual to learn how their tables behave, then move to adaptive once they're comfortable. You can mix modes freely — adaptive on your busiest tables, scheduled elsewhere — and change at any time.
| Operation | What it does |
|---|---|
| File Compaction | Merge small files into optimally-sized data files, resolve deletes |
| Snapshot Expiry | Expire old snapshots and reclaim storage |
| Orphan Cleanup | Remove unreferenced files left by failed jobs or expired snapshots |
| Manifest Rewrite | Consolidate manifest files for faster query planning |
With adaptive mode, LakeOps sequences operations automatically: snapshot expiry runs before compaction (so you never rewrite files about to be garbage-collected), then manifests are consolidated afterward. Tables that need the most help run first.
4. Scale with governance policies
Instead of configuring tables one by one, create governance policies that apply rules across your lake. Go to Manage > Policies > + Create Policy.
Maintenance Policies
Adaptive Maintenance for data-driven automation, or per-operation policies (snapshot expiry, orphan cleanup, manifest rewrites) with cron schedules for workloads that need fixed timing.
Configuration Policies
Enforce table settings — Iceberg format version, file format, write distribution mode — across catalogs and namespaces.
Scope each policy to a catalog, namespace, or specific tables. Per-table policies override broader ones for the same operation type. New tables inherit matching policies automatically.
5. Monitor & refine
Once maintenance is running, track everything from the dashboard:
- •Maintenance tab (per-table) — live debt scores, an action plan showing what is running, cooling, or approaching threshold, and partition-level compaction analysis.
- •Operations Monitoring — fleet-wide adaptive coverage, state distribution (needs action / in progress / cooling / healthy), and tables with the highest ingestion rates.
- •Events — full audit log for every operation with before/after metrics, duration, and status. Available per-table or lake-wide.
- •Insights — resolve automatically as optimizations take effect. New issues surface as workloads change.
Optional — Automate & integrate
Everything above works through the UI. When you're ready to integrate LakeOps into pipelines, CI/CD, or AI workflows, two interfaces are available:
Key guarantees
- ✓No vendor lock-in — works with your existing catalogs, engines, and storage.
- ✓No code or infra changes — connects via metadata. No agents, sidecars, or pipeline rewrites.
- ✓No data movement — reads metadata and writes optimized files back to your storage (S3, GCS, ADLS). Data never leaves your account.