Apache Spark performance tuning, ETL patterns, and integration with Iceberg, Delta Lake, and cloud-native storage.
7 articles
Why Apache Iceberg MERGE INTO gets slow and how to fix it: how a merge scans, joins, rewrites, and commits, the levers that cut the files each upsert touches, and the table maintenance that keeps upserts fast as tables age.
Apache Iceberg 1.12.0 adds Hilbert-curve clustering, a Flink task that converts equality deletes into deletion vectors, V4 manifest readers with relative paths, and vectorized variant reads — while removing Spark 3.4, Flink 2.0, and position deletes carrying row data. Here is what changed and what it means for your tables.
Spark compaction is expensive, slow, and architecturally wrong for file rewrites. Here's how to replace it with purpose-built engines that cost 90% less and finish 95% faster — plus the DIY path if you want to optimize what you already have.
How production Iceberg lakehouses run Spark, Trino, Snowflake, Athena, Flink, and DuckDB on the same tables — covering engine decoupling, write isolation, conflict resolution, catalog coordination, read path optimization, query routing, cross-engine governance, and the control plane that ties it together.
A practical guide to streaming data from Apache Kafka into Apache Iceberg tables — covering Kafka Connect, Apache Flink, Spark Structured Streaming, and CDC with Debezium. Includes configuration examples, schema management, partitioning strategies, production pitfalls, and how to keep streaming tables healthy at scale.
Small files silently degrade every Apache Iceberg lakehouse — inflating S3 costs, slowing query planning, and bloating metadata. This guide covers root causes, measurement, manual and automated fixes, and how to eliminate the problem at scale.
Learn how to implement efficient incremental processing with Apache Iceberg and Spark, including best practices for data lake optimization and performance tuning.
