Event streaming with Apache Kafka — architecture, Kafka Connect, stream processing, and lakehouse integration patterns.
4 articles
A real-time lakehouse is not a faster pipeline. It is three loops running at once — commit, maintain, serve — on open Apache Iceberg tables. This guide covers the freshness budget, why streaming tables decay, and the control plane that keeps them fast enough to serve straight to a UI.
Build a streaming lakehouse on Apache Iceberg — unify Kafka/Flink ingestion and batch analytics without duplicating data. Production patterns, maintenance reality, and how to keep streaming Iceberg tables performant.
Streaming from Kafka into Apache Iceberg creates small files faster than any other write pattern. This guide covers why standard compaction approaches fail for streaming tables, how to measure compaction need, implement partition-aware compaction that avoids writer conflicts, tune rewriteDataFiles parameters, and run maintenance autonomously at scale.
A practical guide to streaming data from Apache Kafka into Apache Iceberg tables — covering Kafka Connect, Apache Flink, Spark Structured Streaming, and CDC with Debezium. Includes configuration examples, schema management, partitioning strategies, production pitfalls, and how to keep streaming tables healthy at scale.
