Deep dives on Apache Iceberg, lakehouse architecture, and running data lakes in production—written by the LakeOps team.
8 articles · Medium, Substack & DEV
Browse articles
QueryFlux is a universal SQL proxy and multi-engine query router in Rust—one access layer in front of Trino, DuckDB, StarRocks, and Athena with routing, dialect translation, and observability.
How we compacted 4.5 TB across 10 real production tables, achieved up to 99.8% file reduction, and made Apache Spark OOM on a job we finished in 11 minutes.
How we built a high-performance, distributed compaction engine for Apache Iceberg using Rust and DataFusion—architecture, design choices, and lessons learned.
See how a 350TB data lake shrank to 230TB in 10 minutes by removing stale data—saving 34% in AWS S3 costs and proving the need for a control plane.
Apache Iceberg delivers speed, but without a control plane snapshots pile up, costs surge, query take more time — starting with expiration.
Unlocking performance vs. optimizing storage — choosing the right compaction strategy for your data lake.
A detailed comparison between Delta Lake and Apache Iceberg, exploring their architectures, performance characteristics, and ideal use cases to help you make the right choice.
Learn how to implement efficient incremental processing with Apache Iceberg and Spark, including best practices for data lake optimization and performance tuning.
