Apache Iceberg
Routing Multiple Query Engines with Iceberg
How to route queries across Trino, Spark, DuckDB, Snowflake, Athena, and Flink on shared Iceberg tables — covering routing architecture, dialect translation, and strategies.
Multi-Engine Query Routing
Route every query to
the right engine
One SQL endpoint. LakeOps sends each query to Trino, Spark, DuckDB, Snowflake, Athena, StarRocks, or Flink based on cost, latency, or workload type. Swap engines without changing application code. Up to 56% lower query spend.
76%Cost reduction
12×Faster queries
95%Faster compaction
100%Autonomous
Open and runs on your stack
Measured Impact
Same SQL, cheaper engine, less wait
Cost reduction
56%lower spend
Workload-aware routing matches queries to optimal pricing models
Latency
4.6×faster
Point lookups on DuckDB vs same query on Athena (0.5s vs 2.3s)
Proxy overhead
0.35msp50
Rust-based routing adds negligible latency to query path
Benchmarks from queryflux-bench suite • Production Iceberg tables • Multi-cloud, multi-engine
How LakeOps optimizes routing
How a query
actually gets routed
Dispatch by query shape, isolate workloads on dedicated endpoints, fail over unhealthy engines, and compact tables so cheaper engines can run more of the SQL.
Query dispatch
Send each query to the engine that fits its shape and price
Interactive lookups go to DuckDB or Trino. Heavy scans go to Spark or Athena. Cost, latency, or throughput strategy is set per routing group. Rules update as query mix changes.
- Three optimization strategies: latency, cost, throughput — per routing group
- Up to 56% cost reduction by routing to the right pricing model
- DuckDB: 0.5s point lookups vs 2.3s on Athena for the same query
Query Routing4 engines active
Trino256 q/h1.8s
Snowflake192 q/h2.1s
AWS Athena128 q/h2.3s
DuckDB64 q/h0.5s
Optimize for: cost · performance · consumer
Routing groups & endpoints
Organize engines by workload — not by team or tool
Group engines into routing endpoints by workload type: storefront analytics on Trino + DuckDB, checkout transactions on Snowflake + StarRocks, catalog ETL on Athena + Spark. Each group has its own policies, capacity limits, and fallback rules — all managed from one console.
- Per-group concurrency limits and queue-based overflow
- Dedicated endpoints per workload (e.g. storefront-analytics.lakeops.dev)
- Add, remove, or swap engines without client changes
Routing groups
Map workloads to engine pools, publish stable URLs, and tune priority without touching clients.
Groups
4
Active
3
Accepting traffic
Paused
1
Inactive groups
Engines
7
Unique in routes
All groups
Edit engines, query types, and published URLs
Analyticsactive
📡 e1fa3c3c.lakeops.dev
Engines: TrinoDuckDB
Query types: SELECTAGGREGATE
Priority: High
📅 06/02/25, 3:20:16 PM
•••
BIactive
Handles BI, DT queries on Transactional databases
📡 1d0e4f1c6.lakeops.dev
Engines: SnowflakeTrino
Query types: INSERTUPDATEDELETE
Priority: Medium
📅 06/02/25, 8:16:52 PM
•••
Data-Team ETLinactive
Batch data transformation and scheduled loads
📡 e11d1ef1.lakeops.dev
Engines: SparkFlink
Query types: BATCHSTREAM
Priority: Medium
📅 06/15/25, 5:40:10 PM
•••
Reportsactive
Routine reporting and dashboard queries
📡 c480dr51.lakeops.dev
Engines: SnowflakeClickHouse
Query types: SELECTVIEW
Priority: Medium
📅 06/02/25, 7:00:38 PM
•••
Engine comparison & health
Side-by-side engine metrics — cost, latency, throughput, success rate
Compare every registered engine across query success rate, average runtime, cost per query, total queries, and data scanned. Spot under-performing engines before they impact SLAs. View engine health, uptime, and resource signals in one place — then route traffic away from degraded backends automatically.
- Real-time comparison: cost/query, avg runtime, data scanned per engine
- Automatic failover when engine health degrades below threshold
- Historical trending to identify capacity bottlenecks before they hit
Engine comparison
Compare engines side-by-side on cost, latency, throughput, and data scanned.
Select engines
Spark
Active
Trino
Active
Athena
Active
Snowflake
Active
DuckDB
Active
Flink
Active
Performance comparison
| Metric | Spark | Trino | Athena | Snowflake |
|---|---|---|---|---|
| Query success rate | 99.2% | 99.5% | 99.9% | 99.8% |
| Average runtime | 3.1s | 1.8s | 2.3s | 2.1s |
| Cost per query | $0.04 | $0.03 | $0.05 | $0.08 |
| Total queries | 3,120 | 2,456 | 1,280 | 1,876 |
| Data scanned | 4.2 TB | 2.8 TB | 1.5 TB | 3.5 TB |
Cost vs latency
Lower-left is ideal
Low cost
High cost
Success rate
Spark
99.2%
Trino
99.5%
Athena
99.9%
Snowflake
99.8%
Table-aware optimization
Compacted tables let DuckDB run queries that used to need Spark
Routing alone improves dispatch — but compacted, sorted tables unlock even more engines for any given query. LakeOps continuously optimizes file layout, manifests, and statistics so that engines like DuckDB and StarRocks can handle queries that previously required heavy distributed compute.
- Compacted tables reduce the minimum viable engine per query shape
- Puffin statistics enable more aggressive file-level pruning across all engines
- Sorted data + lean manifests = more queries eligible for sub-second engines
Table OptimizationRouting-aware
Before optimizationOnly Spark viable
SparkTrinoDuckDBAthena
After optimization4 engines eligible
SparkTrinoDuckDBAthena
One set of rules
Stop maintaining a driver config per engine per team
Set cost ceilings, latency targets, fallbacks, and capacity once. Clients connect to one endpoint. Add or remove backends without touching application code. Every query is logged with engine, cost, and reason.
- Unified policy: cost ceilings, latency targets, priority queues per group
- No N×M driver configurations — clients connect once
- Full audit trail: which query went where, why, and what it cost
Routing PolicyUnified
Interactive BI → low-latency pool
ETL batch → compute-optimized
Ad-hoc exploration → cost-optimized
Streaming ingest → stream-native
Product tour
Engine management from one console
Register engines, compare performance, monitor health, and route traffic — without per-engine scripts or duplicate dashboards.
Open Source
Powered by QueryFlux
QueryFlux is an open-source, Rust-based SQL proxy that provides multi-engine query routing with protocol translation, capacity management, and observability. It works as a standalone proxy — and gains intelligent optimization when connected with LakeOps.
QueryFlux standalone
Deploy QueryFlux as your multi-engine SQL proxy without any other dependencies. One endpoint replaces N×M driver configurations — clients connect once, the backend topology is config, not code.
4 frontend protocols
Trino HTTP, PostgreSQL wire, MySQL wire, Arrow Flight SQL
6+ backend engines
Trino, DuckDB, StarRocks, Athena, ClickHouse, Snowflake, and more
Cost, latency, or throughput per group
Protocol-based, header-based, regex, client tags, Python script logic
SQL dialect translation
sqlglot-backed conversion across 31+ SQL dialects automatically
Per-group capacity & queuing
Concurrency limits, overflow queuing, and fallback routing per cluster group
Observability built in
Prometheus metrics, Grafana dashboards, QueryFlux Studio UI, Admin API
QueryFlux + LakeOps
When connected with LakeOps, QueryFlux gains access to table telemetry, query history, and optimization signals — enabling intelligent routing decisions that go beyond static rules.
Table-health-aware routing
Route queries away from tables with fragmented manifests or high delete file counts to engines that handle degraded state better
Query-pattern learning
LakeOps telemetry feeds historical latency and cost data per query shape — routing improves automatically over time
Optimization-driven engine expansion
As LakeOps compacts and sorts tables, more engines become viable for each query — expanding routing options dynamically
Unified cost model
Combine query routing cost data with compaction compute savings for full lake-wide cost visibility and optimization
Policy-driven automation
Set cost ceilings, latency targets, and priority rules — LakeOps enforces them across routing and table optimization together
Lakehouse Control Plane
Routing is one piece. See the full platform
Routing sits next to compaction, snapshot expiry, table health, and MCP guardrails — same catalogs, same tables.
Table Health Overview786 tables
566
Healthy
105
Warning
70
Critical
92%
Optimized
| Table | NS | Size | Status |
|---|---|---|---|
| customer_orders | orders | 1.24 TB | HEALTHY |
| payment_transactions | payments | 860 GB | WARNING |
| raw_clickstream | analytics | 4.6 TB | CRITICAL |
| product_catalog | products | 42 GB | HEALTHY |
| user_sessions | analytics | 1.9 TB | WARNING |
| inventory_levels | operations | 320 GB | HEALTHY |
| shipping_events | logistics | 580 GB | HEALTHY |
| search_query_logs | analytics | 3.2 TB | CRITICAL |
Observability
Learns from query patterns and table signals to decide what to optimize, when, and how — no schedules, no manual tuning.
- Closed-loop sense → plan → execute → learn
- Zero manual scheduling or threshold tuning
- Adapts to workload changes in real time
The problem
Athena for a point lookup. Trino stuck behind ETL.
Most teams run 2–5 engines. Without a router, dashboards, batch jobs, and ad-hoc SQL share one cluster — or every app hardcodes its own connection string.
Every query hits the same engine regardless of shape
Without routing, all queries — interactive dashboards, heavy ETL, ad-hoc exploration — land on one engine. CPU-heavy joins pay scan-pricing on Athena; selective lookups wait behind batch jobs on Trino.
N×M driver configurations fragment your platform
Each engine requires its own connection strings, credentials, and client libraries. Every team maintains a separate config per engine — adding or removing a backend means coordinating across every consumer.
No cost awareness in query dispatch
Compute-priced backends (Trino, StarRocks) charge for CPU-seconds. Scan-priced backends (Athena, BigQuery) charge for bytes read. Without cost-aware routing, every query pays the wrong pricing model.
SLA violations when workloads compete on a shared cluster
A batch ETL job competing with an interactive dashboard on the same Trino cluster degrades both. Without workload isolation at the routing layer, latency-sensitive queries suffer unpredictably.
Resources
Learn more
Data Platforms
Apache Iceberg Multi-Engine Architecture
How production Iceberg lakehouses run Spark, Trino, Snowflake, Athena, Flink, and DuckDB on the same tables — covering write isolation, conflict resolution, and query routing.
FinOps
State of Iceberg FinOps and Cost Reduction in 2026
Where lakehouse spend leaks, what to measure, how autonomous management and query routing are replacing manual maintenance — and a practical survey of tools.
Connect in minutes
- no vendor lock-in
1
Connect catalogs & engines
Only metadata is processed — never retained or stored.
2
Get visibility & insights
Telemetry reveals table health and actions needed.
Table health scores
Optimization opportunities
Cost & performance insights
3
Choose your mode
Autopilot, manual approval, or policy-driven.
Autopilot
Manual
Policies
4
Lakehouse optimized
Queries 10x faster
Cost down 76%
Engines optimized
AIs managed
Tables healthy
Fully governed
No vendor lock-in
No code / infra changes
No data changes
Set up in 10 minutes · Works with your existing stack
See routing in action on your workloads
Connect your engines and get a routing analysis in minutes — see which queries should move, how much you'd save, and which engines to add or consolidate.