Real-time and event-driven data engineering — streaming ingestion, CDC pipelines, and near-real-time analytics.
10 articles
Iceberg V3 brings row-level lineage, default column values, and deletion vectors — the features streaming pipelines need for governance without downtime. How V3 changes the streaming governance story on Flink, Kafka, and CDC sources.
Hot/cold data tiering on Apache Iceberg — StarRocks for sub-second dashboards, Iceberg for petabyte history, one SQL surface via federation. Ingestion, tier transitions, dedup, and keeping the cold tier fast.
LinkedIn runs Iceberg CDC on 10,000+ tables — billions of upserts daily, 2+ PB throughput. Equality vs position deletes, delete-file compaction, budgeted maintenance, and WAP branching lessons for any team running MERGE INTO at scale.
Every concurrent write to an Apache Iceberg table risks a commit conflict. This guide covers how Iceberg's optimistic concurrency works, what triggers CommitFailedException, the common conflict scenarios in streaming and maintenance workloads, and the strategies — from partition isolation to branch-based writes — that eliminate conflicts in production.
Streaming from Kafka into Apache Iceberg creates small files faster than any other write pattern. This guide covers why standard compaction approaches fail for streaming tables, how to measure compaction need, implement partition-aware compaction that avoids writer conflicts, tune rewriteDataFiles parameters, and run maintenance autonomously at scale.
A practical guide to streaming data from Apache Kafka into Apache Iceberg tables — covering Kafka Connect, Apache Flink, Spark Structured Streaming, and CDC with Debezium. Includes configuration examples, schema management, partitioning strategies, production pitfalls, and how to keep streaming tables healthy at scale.
Apache Iceberg 1.11.0 lands V3 maturity with production-ready deletion vectors, a native Variant type for semi-structured data, server-side scan planning, built-in table encryption, and a pluggable File Format API that opens the door to next-generation storage formats.
Flink streaming into Iceberg creates thousands of small files per hour. This guide covers checkpoint tuning, write distribution modes, Flink SQL patterns, and why external maintenance is essential for production streaming tables.
Delete files let Iceberg avoid rewriting data on every UPDATE or DELETE — but every unresolved delete file forces readers to reconcile at query time. A deep guide to position deletes, equality deletes, measuring overhead, and resolving accumulation before it tanks performance.
Small files silently degrade every Apache Iceberg lakehouse — inflating S3 costs, slowing query planning, and bloating metadata. This guide covers root causes, measurement, manual and automated fixes, and how to eliminate the problem at scale.
