Faster Queries
12× faster queries
without changing a line of SQL
Query-aware compaction sorts files by the columns your queries actually use. Engine routing picks the fastest runtime. Half the data scanned, every query faster.
LakeOpsLast 30 days Optimization Activity
Key Metrics
Storage
-56% reclaimedCPU
-76% reductionRecent Operations
| Operation | Table | Duration | Impact | Time | Status |
|---|---|---|---|---|---|
| Compact Data Files | customer_orders orders | 4s | 1.24 TB, 16 → 1 files | 57 minutes ago | SUCCESS |
| Expire Snapshots | payment_transactions payments | 27s | 8.2 TB | 4 hours ago | SUCCESS |
| Expire Snapshots | inventory_snapshots_20250702 warehouse | 3s | 2.1 TB | 4 hours ago | SUCCESS |
| Rewrite Manifests | raw_clickstream analytics | 1.9s | 3 → 1 manifests | 5 hours ago | SUCCESS |
| Compact Data Files | product_catalog products | 6m 11.3s | 3,008 → 1,256 files | 6 hours ago | SUCCESS |
Lake Events
Open and runs on your stack
The problem
Four layers of latency
hidden in your Iceberg tables
Poor file layout, file fragmentation, wrong engine choice, and metadata bloat compound silently. Queries slow down as tables grow — not because of data volume, but because of table health.
Unsorted data files
Parquet min/max ranges overlap across every file. The engine can’t skip anything — every query scans nearly everything.
Thousands of tiny files
Streaming ingestion creates a new file per micro-batch. Listing, opening, and reading thousands of small files dominates query time.
Wrong engine for the query
A 50 MB ad-hoc scan on a Spark cluster wastes minutes spinning up executors. A 500 GB join on DuckDB runs out of memory.
Stale table metadata
Manifest files accumulate with every write. Planning overhead grows linearly — adding seconds before any data is even read.
How it works
From full table scans to surgical reads
LakeOps fixes query performance at every layer — data layout, file consolidation, engine routing, and metadata — each one amplifying the last.
01 · Query-aware sort
Sort files by the columns your queries actually use
LakeOps collects WHERE, JOIN, and GROUP BY column frequency from every connected engine — Trino, Spark, Snowflake, Athena, DuckDB — then physically re-sorts data files so Parquet min/max pruning skips entire file groups. The sort order adapts as query patterns evolve.
- Cross-engine telemetry — learns from every engine, not just one
- 51% less data scanned per query after sort optimization
- Sort order evolves automatically as query patterns shift
Columns ranked by query frequency
customer_idevent_dateproduct_idregionData scanned
-51%
per query
Query speed
12×
faster
02 · Right-sized files
Merge small files into optimally-sized blocks
Streaming writes fragment tables into thousands of tiny files. LakeOps merges them into right-sized files using a purpose-built Rust engine that processes at 2,500+ MB/s — 95% faster and 90% cheaper than Spark.
- 970 → 87 files on a typical table — 62% fewer S3 GETs per query
- Rust + DataFusion pipeline — no JVM, no GC, no OOM
- Bounded memory with disk spill — handles multi-TB tables safely
Files
970
87
Throughput
—
2,522 MB/s
S3 GETs
100%
-62%
03 · Engine routing
Route each query to the fastest engine for the job
One SQL endpoint. LakeOps profiles each query’s shape, data volume, and latency target, then routes it to the engine that delivers the fastest result — Trino for sub-second interactive scans, Spark for heavy ETL, DuckDB for lightweight ad-hoc.
- One endpoint, every engine — no connection string juggling
- Cost and latency targets per query class
- Automatic fallback if the primary engine is saturated
Engine comparison
Compare engines side-by-side on cost, latency, throughput, and data scanned.
Select engines
Performance comparison
| Metric | Spark | Trino | Athena | Snowflake |
|---|---|---|---|---|
| Query success rate | 99.2% | 99.5% | 99.9% | 99.8% |
| Average runtime | 3.1s | 1.8s | 2.3s | 2.1s |
| Cost per query | $0.04 | $0.03 | $0.05 | $0.08 |
| Total queries | 3,120 | 2,456 | 1,280 | 1,876 |
| Data scanned | 4.2 TB | 2.8 TB | 1.5 TB | 3.5 TB |
Cost vs latency
Lower-left is ideal
Success rate
04 · Metadata optimization
Keep planning overhead near zero as tables grow
Every Iceberg write adds manifest entries. LakeOps rewrites manifests to merge redundant entries, drops expired snapshots, and prunes metadata so query planning stays fast — even on tables with millions of files.
- Manifest rewrite — fewer entries, faster planning
- Snapshot expiry — only keep what retention requires
- Sub-second planning on tables with 100K+ files
| Table | NS | Size | Status |
|---|---|---|---|
| customer_orders | orders | 1.24 TB | HEALTHY |
| payment_transactions | payments | 860 GB | WARNING |
| raw_clickstream | analytics | 4.6 TB | CRITICAL |
| product_catalog | products | 42 GB | HEALTHY |
| user_sessions | analytics | 1.9 TB | WARNING |
| inventory_levels | operations | 320 GB | HEALTHY |
| shipping_events | logistics | 580 GB | HEALTHY |
| search_query_logs | analytics | 3.2 TB | CRITICAL |
Results
Cut costs and boost performance
Benchmarks from production-grade tables across multiple engines and clouds.
Query speed
After compaction + layout optimization
CPU reduction
Compute hours across all engines
Storage saved
Orphans, snapshots & bloat removed
Table health
Autonomous maintenance keeps every table optimized
Connect in minutes
- no vendor lock-in
Connect catalogs & engines
Only metadata is processed — never retained or stored.
Get visibility & insights
Telemetry reveals table health and actions needed.
Choose your mode
Autopilot, manual approval, or policy-driven.
Lakehouse optimized
Built for enterprise
grade data lakes
SOC 2, SSO, RBAC, dedicated support, and the scale your largest Iceberg lakes demand.
Security & compliance
SOC 2 Type II, encryption, SSO/RBAC, and audit trails for regulated teams.
Scale & control
One control plane for your full lake. Real-time visibility, policies, and predictable performance.
Support & training
Dedicated onboarding, training, and enterprise SLAs. Deploy in VPC or on-prem.
Get started
See LakeOps on your stack
Get a personalized walkthrough with your data and architecture.
Short call, no commitment.
Typically 30 min · Free