Guides and insights from the LakeOps team on Apache Iceberg,lakehouse architecture, and production operations.
Iceberg tables degrade silently — small files pile up, snapshots bloat metadata, and query latency creeps higher. A breakdown of the nine components every production data lake needs to stay healthy.
Netflix spent years building an intelligent lakehouse — Polaris, Autotune, janitors, and Metacat. LakeOps lets every team build the same — and go beyond — in minutes.
How to route queries across Trino, Spark, DuckDB, Snowflake, Athena, and Flink on shared Iceberg tables — SQL routing proxy, dialect translation, and table-aware optimization.
A deep guide to bin-pack, sort, and Z-order compaction strategies for Apache Iceberg — when to use each, how to configure them, and how to automate strategy selection across hundreds of tables.
How to choose, evaluate, and evolve partitioning strategies for Apache Iceberg tables — decision frameworks for real workloads, partition lifecycle management, and when to change your strategy.
How STACKIT built a managed lakehouse offering on Kubernetes — custom operators, CRD-driven provisioning, multi-tenant Iceberg catalogs, and the engineering lessons from bringing a sovereign lakehouse service to market in Europe.
Running Iceberg at 10 tables is configuration. Running it at 10,000 is infrastructure. Production lessons on infrastructure evolution, Parquet tuning, Spark configuration, catalog scaling, enterprise security, and observability-driven optimization for production Iceberg deployments.
AI workloads running against Iceberg tables on S3 need more than fast queries — they need provably secure, least-privilege access to every byte they touch. This article walks through a zero-trust data architecture built on vended credentials, the Iceberg REST catalog, and Kubernetes-native orchestration — replacing static keys with short-lived, table-scoped tokens enforced at the storage layer.
AI agents fail in production because they are overwhelmed with data but starved for context. The bottleneck is not the model — it is the data stack. Apache Iceberg turns lakehouse storage into a live, versioned context layer that powers structured RAG, schema-aware agents, and governed reasoning grounded in truth.
