Streaming Lakehouse on Apache Iceberg: Kafka, Flink, and Real-Time Pipelines Without Duplication

The streaming lakehouse on Apache Iceberg promises one table format for both real-time and historical analytics — but most teams still run two parallel systems. Kafka (or Kinesis, or Pulsar) holds the real-time copy. Iceberg holds the analytical copy. The data is the same; the systems are not. And the operational cost of keeping both consistent is where data platform teams spend an outsized share of engineering time.

This is the duplication problem. It is not a theoretical concern. It is the daily reality of any team running streaming analytics at scale: two storage systems, two sets of schemas, two consistency models, two retention policies, two cost centers — for one logical dataset.

Streaming data into Iceberg is easy. The tooling exists: Flink, Kafka Connect, Spark Structured Streaming. You can have events landing in Iceberg tables within an afternoon. Keeping those streaming tables healthy is the hard part. Every Flink checkpoint creates a batch of small Parquet files. Every minute adds a new Iceberg snapshot. Within hours, a streaming table can accumulate thousands of tiny files and hundreds of snapshots, and query performance degrades from seconds to minutes. Without continuous, automated maintenance — compaction, snapshot expiration, orphan file cleanup — streaming on Iceberg is a ticking time bomb.

LakeOps addresses this directly — a control plane for Apache Iceberg lakehouses that runs continuous, event-driven compaction, snapshot expiration, and orphan cleanup on streaming tables, keeping them healthy without custom maintenance scripts or manual scheduling.

This article explores the streaming lakehouse architecture — what it is, why it matters, and where the ecosystem is heading. It covers production patterns from Flink, Kafka Connect, and Spark Structured Streaming on Iceberg today, plus the emerging Apache Fluss project and its approach to unifying streaming and batch access.

The Duplication Problem

The standard modern data architecture looks like this: application services emit events to Kafka. A streaming layer — usually Flink — consumes those events for real-time processing: fraud detection, session windowing, feature engineering, real-time dashboards. Separately, a batch ingestion pipeline — sometimes the same Flink job, sometimes a different Spark job, sometimes a Kafka Connect sink — lands those events into Iceberg tables on S3 or GCS for analytical workloads: BI, ad-hoc SQL, ML training, compliance reporting.

This means the same event exists in two places. Kafka retains it for hours or days (sometimes longer with tiered storage). Iceberg retains it for months or years. The schemas may drift between the two. The ordering guarantees differ. The query semantics are completely different — Kafka is append-only, keyed, partitioned by topic; Iceberg is columnar, partitioned by time or business key, queryable by SQL engines.

The consequences compound:

• Storage cost doubles. Every event is stored twice — once in Kafka's segment format, once in Parquet on object storage. For high-volume streams (clickstream, telemetry, IoT), this is measured in petabytes.
• Consistency is manual. If a Flink job fails and replays from a Kafka offset, does the Iceberg table get duplicates? If a late-arriving event hits Kafka after the batch window closed, does it appear in the lake? These are solvable problems, but each solution is custom, fragile, and different per pipeline.
• Schema evolution is disconnected. Kafka schemas evolve in the Schema Registry. Iceberg schemas evolve via ALTER TABLE. There is no single source of truth. A field added in Kafka may not propagate to Iceberg for days if the sink connector does not handle it.
• Latency is a cliff, not a gradient. Data is either in Kafka (real-time, but hard to query with SQL) or in Iceberg (queryable, but minutes to hours stale). There is no smooth continuum. You cannot run the same SQL query that spans both real-time and historical data without custom federation layers.
• Operational burden multiplies. Kafka needs broker management, partition rebalancing, consumer group monitoring. Iceberg needs compaction, snapshot expiration, metadata management. Each system has its own failure modes. Debugging a data quality issue requires correlating state across both.

This is not a niche problem. It is the default architecture at most companies running streaming analytics. And it has persisted because, until recently, there was no credible alternative that handled both real-time and historical workloads in a single system without unacceptable tradeoffs.

The Streaming Lakehouse Vision

The streaming lakehouse is a simple idea with hard engineering: make the table the core abstraction for both real-time and historical data access. One table. One schema. One storage layer. Data is written continuously — as a stream — and is immediately available for both low-latency consumption (subscribe to new rows as they arrive) and analytical queries (scan historical data with full SQL).

This is not the same as "just write faster to Iceberg." The streaming lakehouse requires the table format to natively support two access patterns that have historically been separate:

• Row-level streaming reads. Consumers can subscribe to a table and receive new rows as they are committed, with ordering guarantees and offset tracking — the same semantics Kafka provides today.
• Columnar analytical reads. Query engines can scan the same table with predicate pushdown, column pruning, and partition elimination — the same semantics Iceberg provides today.

In a true streaming lakehouse, there is no separate message broker. The table itself is the stream. Producers write to the table. Streaming consumers tail the table. Analytical queries scan the table. The data exists once.

The benefits are structural:

• Single storage cost. Data is stored once, in a format that serves both streaming and analytical workloads.
• Transactional consistency. Every write is an atomic commit to the table. Streaming consumers and analytical queries see the same consistent state.
• Unified schema. One schema definition governs all access patterns. Schema evolution is applied once and visible everywhere.
• Latency as a dial, not a cliff. The same query engine can read the latest committed rows (sub-second freshness) or scan years of historical data. The access pattern is the same.
• Single operational surface. One system to monitor, tune, and maintain — not two.

This vision has been articulated by multiple projects and vendors, but the implementation details vary significantly. The question is not whether the streaming lakehouse is desirable — it clearly is — but whether the technology exists to build one without unacceptable compromises on latency, throughput, or query performance.

Apache Fluss: Streaming Native in the Lakehouse

Apache Fluss (incubating) is the most direct attempt to build a streaming lakehouse from first principles. Fluss is designed to make the table handle row-level, columnar, and vector data natively — eliminating the need for separate streaming and batch systems.

Fluss is not a connector between Kafka and Iceberg. It is a replacement for Kafka in the streaming lakehouse context — providing the real-time ingestion layer while tiering data to lakehouse formats (Paimon, Iceberg, or Lance) for analytical storage. The core idea: a single storage system that supports real-time row-level writes and reads (like Kafka) and analytical columnar scans (via tiered lakehouse storage), with the table as the unified abstraction.

How Fluss Works

Fluss introduces a dual storage model within a single table:

• Log storage handles real-time writes and streaming reads. Data arrives as individual rows, stored in Apache Arrow columnar format for sub-second analytical access. Rows are durably persisted with ordering guarantees and are immediately available for streaming consumers. This is the Kafka-equivalent layer — but columnar rather than row-based.
• Lake storage handles analytical queries. A tiering service continuously compacts log data into Parquet files on object storage, organized by partition, with Iceberg or Paimon-compatible metadata. This is the Iceberg-equivalent layer.

The critical difference from the Kafka+Iceberg stack is that these two storage layers are not separate systems connected by a pipeline. They are two tiers of the same table, managed by a single system, with a single schema and unified metadata. The tiering service continuously moves data from log to lake with configurable freshness (default 3 minutes), keeping the two layers synchronized.

When a producer writes a row to a Fluss table, it is immediately available for streaming consumers via the log. In the background, the tiering service compacts log data into columnar files on the lake. An analytical query can use Union Reads to access both — the latest rows from the log that have not yet been tiered, plus the historical columnar data from the lake — and merge them into a single consistent result set.

What Fluss Changes

For teams currently running Kafka + Flink + Iceberg, Fluss eliminates the middle layer. There is no need for a Flink job whose only purpose is to shuttle data from Kafka topics to Iceberg tables. The table itself handles ingestion, retention, and serving.

This has several practical implications:

• No checkpoint-driven file creation. Flink's Iceberg sink creates a new set of files on every checkpoint. Fluss manages its own compaction lifecycle, producing optimally-sized files independent of the producer's write cadence.
• No snapshot explosion. Instead of one Iceberg snapshot per checkpoint (potentially every 30 seconds), Fluss controls when lake-side snapshots are created based on data volume, not writer behavior.
• No schema synchronization. The schema is defined once on the Fluss table. Streaming producers and analytical consumers see the same schema. No Schema Registry, no Iceberg ALTER TABLE, no drift.
• No dual retention management. Kafka retention and Iceberg snapshot expiration are replaced by a single table-level retention policy.

Fluss is still in Apache incubation and is not production-ready for most workloads. But it represents the direction the ecosystem is moving: tables that natively support both streaming and analytical access patterns, with the storage engine managing the complexity that today falls on the operator.

Current Streaming Patterns on Iceberg

While the streaming lakehouse is the destination, most teams today are building on what exists: Apache Flink, Kafka Connect, and Spark Structured Streaming writing into Iceberg tables. Each approach works. Each has tradeoffs that matter in production.

Flink CDC with Checkpointing

Apache Flink is the most common streaming writer for Iceberg. The pattern is straightforward: a Flink job consumes from Kafka (or directly from a database via CDC), performs any necessary transformations, and writes to an Iceberg table using the IcebergSink. Commits happen at Flink checkpoint boundaries.

1FlinkSink.forRowData(input)2    .tableLoader(TableLoader.fromHadoopTable(tablePath))3    .overwrite(false)4    .distributionMode(DistributionMode.HASH)5    .writeParallelism(16)6    .build();

The checkpoint interval controls both data freshness and file creation rate. A 30-second checkpoint means data is visible in Iceberg within 30 seconds of arrival — but also means 2 new data files per task per minute. With 16 write tasks, that is 32 files per minute, 1,920 files per hour, 46,080 files per day — per table.

For CDC workloads, Flink supports MERGE INTO operations via equality delete files. This enables upserts and deletes, but adds additional file overhead: each merge operation produces both data files and delete files, both of which must be compacted.

Flink-Iceberg integration is mature and handles exactly-once semantics correctly via checkpoint coordination. The challenge is entirely operational: managing the file and snapshot accumulation that continuous checkpointing produces.

Kafka Connect Iceberg Sink

The Kafka Connect Iceberg Sink Connector is the simplest path for append-only workloads. It reads from Kafka topics and writes Parquet files to Iceberg tables on a configurable commit interval (default: 5 minutes).

Because Kafka Connect batches data between commits, it produces fewer, larger files than Flink — making it gentler on table health. The tradeoff is latency: data is not visible until the next commit, and commit intervals shorter than 1–2 minutes are not recommended due to the overhead of Iceberg metadata operations.

Kafka Connect does not support upserts, deletes, or any transformation beyond basic SMTs. It is strictly an append pipeline. For teams that need to land events in Iceberg without transformation and can tolerate minute-level latency, it is the lowest-maintenance option.

Spark Structured Streaming

Spark Structured Streaming writes to Iceberg via micro-batch triggers. Each trigger reads available data from the source, processes it, and commits a batch to the Iceberg table.

1df.writeStream \2    .format("iceberg") \3    .outputMode("append") \4    .trigger(processingTime="2 minutes") \5    .option("checkpointLocation", checkpoint_path) \6    .toTable("catalog.db.events")

The trigger interval plays the same role as Flink's checkpoint interval: it controls freshness and file creation rate. Spark's micro-batch model naturally produces fewer, larger files than Flink's per-checkpoint writes, but at the cost of higher latency — typical trigger intervals are 1–5 minutes.

Spark Structured Streaming supports foreachBatch for complex write patterns, including upserts via MERGE INTO. This is commonly used for CDC and slowly changing dimension workloads:

1def upsert_to_iceberg(batch_df, batch_id):2    batch_df.createOrReplaceTempView("updates")3    spark.sql("""4        MERGE INTO catalog.db.customers t5        USING updates s6        ON t.customer_id = s.customer_id7        WHEN MATCHED THEN UPDATE SET *8        WHEN NOT MATCHED THEN INSERT *9    """)10 11df.writeStream \12    .foreachBatch(upsert_to_iceberg) \13    .trigger(processingTime="5 minutes") \14    .start()

The advantage of Spark is ecosystem: most data teams already have Spark clusters, and Spark's DataFrame API is familiar. The disadvantage is that Spark is not designed for low-latency streaming — sub-minute freshness is difficult to achieve reliably.

Tradeoffs Across Approaches

All three approaches share a fundamental characteristic: they are batch writers operating on a streaming schedule. Flink checkpoints, Kafka Connect commits, and Spark micro-batches all produce discrete batches of files at regular intervals. The faster the interval, the more files, the more snapshots, and the more maintenance required.

This is not a bug — it is a consequence of writing to a batch-oriented format (Parquet files + Iceberg metadata) from a streaming source. The streaming lakehouse projects (Fluss, Paimon) aim to eliminate this impedance mismatch. But for teams building on Iceberg today, understanding and managing these tradeoffs is the core operational challenge.

The Operational Reality of Streaming Writes

The gap between "streaming into Iceberg works" and "streaming into Iceberg works in production" is almost entirely about file management. Every streaming writer, regardless of engine, creates files on a schedule dictated by the writer's commit cadence — not by what is optimal for the table.

Small File Accumulation

A Flink job with a 1-minute checkpoint interval and 8 write tasks creates at least 8 data files per minute. Over 24 hours, that is 11,520 files. Over a week, 80,640. These files are small — often 1–10 MB each — a classic small files problem because the checkpoint interval does not allow enough data to accumulate for optimally-sized files (128–512 MB is typically ideal for Iceberg).

The impact is direct: query engines must open, read metadata for, and scan each file individually. A query that scans 100 well-sized files takes seconds. The same query scanning 80,000 small files takes minutes — even though the total data volume is identical.

This is not hypothetical. It is the default outcome of any streaming Iceberg pipeline that runs for more than a few hours without compaction.

Snapshot Explosion

Every commit to an Iceberg table creates a new snapshot. Snapshots are valuable — they enable time travel and rollback. But streaming writers create snapshots at their commit cadence: every 30 seconds, every minute, every 5 minutes. A table with a 1-minute commit interval accumulates 1,440 snapshots per day, 10,080 per week.

Each snapshot references a manifest list, which references manifest files, which reference data files. The metadata tree grows with every snapshot. Query planning — the process of determining which files to read — must traverse this metadata. With thousands of snapshots, planning time alone can exceed the actual query execution time.

Snapshot expiration is not optional for streaming tables. It is a survival requirement. LakeOps handles this automatically, expiring snapshots based on configurable age and count thresholds, with awareness of active readers to avoid removing snapshots that are still in use.

The Compounding Effect

Small files and snapshot accumulation are not independent problems — they compound. More snapshots mean more manifest files. More manifest files mean more metadata to scan during planning. More small files mean more splits during execution. The degradation is super-linear: a table that takes 2 seconds to query on day one can take 30 seconds by day three and 5 minutes by day seven — with no increase in data volume.

This is why streaming tables need continuous, automated maintenance — not scheduled weekly jobs. The maintenance cadence must match the write cadence. If you write every minute, you need compaction running continuously, not nightly.

How Paimon and Fluss Differ from Kafka+Flink+Iceberg

The current dominant stack — Kafka for ingestion, Flink for processing, Iceberg for storage — is a pipeline architecture. Data moves between three separate systems, each with its own storage format, retention model, and operational requirements. Apache Paimon and Apache Fluss represent two different approaches to collapsing this stack.

Apache Paimon: Streaming-First Table Format

Paimon is a table format designed for streaming workloads from the start — unlike Iceberg and Delta, which were designed for batch and later added streaming support. The key difference is that Paimon manages its own compaction and merge-on-read semantics natively within the format.

When Flink writes to a Paimon table, data lands in a log-structured merge tree (LSM). Reads merge across levels of the LSM in real-time, so queries always see the latest data without waiting for compaction to complete. Paimon's compaction runs as a background process within the same system, producing optimally-sized files without operator intervention.

This eliminates the two biggest problems with Flink+Iceberg: checkpoint-driven file creation and manual compaction scheduling. Paimon absorbs streaming writes at any cadence and manages the physical layout internally.

The tradeoff is ecosystem: Paimon tables are primarily consumed via Flink and Spark, with a separate Trino connector available through the apache/paimon-trino project. They are not natively readable by Athena or Snowflake. For teams that need broad engine compatibility across all major cloud and query services, Iceberg remains the safer choice.

Apache Fluss: Replacing the Message Broker

While Paimon replaces Iceberg in the storage layer, Fluss goes further — it replaces Kafka as the ingestion layer and manages the tiering to lakehouse formats internally. Fluss provides the real-time log (replacing Kafka) and tiers data to Paimon or Iceberg for analytical access — all within a single system.

The distinction matters architecturally. With Paimon, you still need Kafka to receive events from application services. Paimon replaces the storage format, but the ingestion path remains Kafka → Flink → Paimon. With Fluss, the ingestion path is Application → Fluss, with the table itself serving as the streaming buffer and the tiering service moving data to Paimon or Iceberg for long-term analytical access.

For existing architectures with established Kafka deployments, Paimon is a more incremental change — swap the table format, keep the pipeline. For greenfield architectures or teams willing to replace the entire stack, Fluss is the more radical simplification.

The Common Thread

Both Paimon and Fluss share a design principle: the storage system should manage its own physical layout in response to incoming data, rather than relying on external writers and external maintenance jobs. This is the fundamental shift from the current Kafka+Flink+Iceberg stack, where the table format (Iceberg) is a passive recipient of whatever files the writer (Flink) produces, and maintenance (compaction, expiration) is a separate operational concern.

Whether the future is Paimon, Fluss, or a converged Iceberg with native streaming support, the direction is clear: tables that actively manage themselves in response to streaming writes, rather than tables that passively accumulate whatever writers dump into them.

The Maintenance Burden: Streaming Tables Need 10x More Care

This is the point most teams discover too late: streaming Iceberg tables require fundamentally more maintenance than batch tables. The ratio is not 2x — it is closer to 10x. And the consequences of falling behind are not gradual — they are cliff-like.

Why 10x

A batch table that loads once per day needs compaction once per day (or less). It creates one snapshot per load. Its file count grows linearly with data volume. Maintenance can be a nightly cron job that runs after the load completes.

A streaming table with a 1-minute commit interval creates 1,440 snapshots per day. It creates 8–64 data files per minute (depending on parallelism). Its file count grows linearly with time, not data volume — even during low-traffic periods, every checkpoint produces files. Within a single day, a streaming table can have more files and more snapshots than a batch table accumulates in a month.

Compaction must run continuously — not nightly, not hourly, but as a background process that keeps pace with the write rate. Snapshot expiration must run frequently enough to prevent metadata bloat. Orphan file cleanup must handle the abandoned files from failed checkpoints, which are more common in streaming jobs that run 24/7.

What Happens When You Fall Behind

When compaction falls behind on a streaming table, the effects cascade:

• Query latency increases. More files to scan means more I/O, more split planning, more task scheduling overhead. Queries that took 3 seconds now take 30.
• Planning time dominates. Iceberg's query planning must enumerate all manifests and apply partition filters. With thousands of manifests from thousands of snapshots, planning alone can take 10+ seconds — before any data is read.
• Metadata operations slow down. Every new commit must read the current metadata, which includes the full manifest list. As the manifest list grows, commit latency increases, which can cause Flink checkpoints to time out.
• Compaction itself gets harder. More small files means each compaction job must process more inputs. If compaction is slow enough that new files arrive faster than old files are compacted, the table enters a death spiral.
• Storage costs spike. Thousands of unreferenced files from expired snapshots and abandoned checkpoints accumulate. Without orphan file cleanup, storage costs grow indefinitely.

Most teams discover this after their first production streaming table has been running for a few days. The table "worked" in development (where it ran for hours) but degrades rapidly in production (where it runs for weeks).

LakeOps for Streaming Tables

This is where LakeOps becomes critical. LakeOps provides event-driven compaction triggers based on file count thresholds — when a streaming table crosses a configurable file count, compaction kicks in automatically. There is no fixed schedule to tune. The system responds to actual table state.

Key capabilities for streaming workloads:

• Adaptive scheduling. LakeOps detects streaming tables (based on commit frequency) and schedules compaction more aggressively than for batch tables. A table receiving commits every minute gets compacted continuously; a table loaded daily gets compacted daily.
• Snapshot expiration. Streaming tables create thousands of snapshots per day. LakeOps automates snapshot expiration based on age and count thresholds, keeping metadata lean without manual intervention.
• Conflict-aware operations. LakeOps never conflicts with active streaming writers. Compaction and maintenance operations use Iceberg's optimistic concurrency control, retrying automatically if a conflict occurs with a concurrent writer commit.
• Real-time observability. LakeOps provides visibility into file accumulation rates, snapshot counts, and table health metrics — so you can see when a streaming table is trending toward degradation before queries start failing.
• Orphan file cleanup. Failed Flink checkpoints leave behind partially-written files that are never referenced by any snapshot. LakeOps identifies and removes these orphan files automatically, preventing unbounded storage growth.

Without this kind of autonomous maintenance, operating streaming Iceberg tables requires a dedicated team writing custom compaction jobs, scheduling snapshot expiration scripts, and monitoring file counts manually. LakeOps eliminates that toil. See how it works for specific streaming patterns: Kafka to Iceberg compaction, Flink Iceberg optimization.

Production Patterns for Streaming Iceberg Tables

For teams operating on the current Kafka+Flink+Iceberg stack (which is most teams), there are concrete patterns that reduce the maintenance burden and improve table health. None of these eliminate the need for compaction — they reduce the rate at which small files accumulate.

Checkpoint Interval Tuning

The single most impactful knob is the Flink checkpoint interval. Most teams default to 30 seconds or 1 minute. For Iceberg workloads, this is almost always too aggressive.

The right checkpoint interval depends on the latency SLA. If downstream consumers need data within 1 minute, a 1-minute checkpoint is necessary — but the maintenance cost is high. If 5 minutes is acceptable, a 5-minute checkpoint reduces file creation by 5x with no architectural change.

1execution.checkpointing.interval: 300000  # 5 minutes2execution.checkpointing.min-pause: 60000   # at least 1 minute between checkpoints3execution.checkpointing.timeout: 600000    # 10 minute timeout

For tables that do not serve real-time dashboards — staging tables, feature stores, compliance archives — checkpoint intervals of 10–15 minutes are often appropriate. The file creation rate drops from thousands per day to hundreds.

Commit Coalescing

Some streaming frameworks support coalescing multiple writer commits into a single Iceberg commit. Instead of each Flink subtask producing an independent file on every checkpoint, a coordinator collects all files and issues a single commit with all of them.

Flink's Iceberg sink already does this to some degree — all files from a single checkpoint are committed atomically in one snapshot. The opportunity for further coalescing is to buffer multiple checkpoints before committing to Iceberg, trading latency for fewer snapshots. This is supported in some custom implementations but is not a standard Flink feature.

Write-Side Buffering

Instead of writing every checkpoint's data directly to Parquet files on S3, some pipelines buffer data in local storage (or a staging area) and flush to Iceberg only when sufficient data has accumulated to produce well-sized files.

This is the approach that Paimon takes natively (via its LSM tree), but it can be approximated on Iceberg by using Flink's state backend to accumulate records and flush them in larger batches. The tradeoff is increased state size and higher risk of data loss during failures (since buffered data must be replayed from the checkpoint).

Partition-Aware Writes

Streaming writes that are not partition-aware create files across all active partitions on every checkpoint. If a table is partitioned by day, and you receive late-arriving data for the last 7 days, every checkpoint creates files in 7 partitions — multiplying the small file problem by 7.

Partition-aware writes route records to the correct partition before writing, ensuring that each checkpoint only creates files in partitions that received data. Flink's distributionMode(DistributionMode.HASH) enables this by shuffling records by partition key before writing:

1FlinkSink.forRowData(input)2    .tableLoader(tableLoader)3    .distributionMode(DistributionMode.HASH)4    .writeParallelism(16)5    .build();

This reduces the number of files per checkpoint but increases network shuffling. For high-volume streams, the tradeoff is almost always worth it.

Continuous Background Maintenance

The most important production pattern is not about writing — it is about maintenance. Streaming tables need continuous background compaction running in parallel with the streaming writer. This is not a periodic job. It is a permanent process.

The compaction process must:

• Run at a cadence that matches or exceeds the write rate
• Target partitions with the highest file counts first
• Produce files in the optimal size range (128–512 MB)
• Handle conflicts with concurrent streaming commits gracefully
• Expire old snapshots without removing data still referenced by active queries
• Clean up orphan files from failed writer commits

This is exactly what LakeOps automates. Rather than building and maintaining custom compaction infrastructure, LakeOps provides a managed control plane that continuously monitors streaming tables and executes maintenance operations based on real-time table health — not fixed schedules. For teams running streaming Iceberg tables, it is the difference between a pipeline that works and a pipeline that stays working.

For more on autonomous table maintenance and its impact on streaming workloads, see automating Iceberg table maintenance and the managed Iceberg solutions overview.

Where the Ecosystem Is Heading

The streaming lakehouse is not a future concept — it is an active engineering effort across multiple projects. The trajectory is clear:

Iceberg is adding streaming-native features. Iceberg's row-level deletes (v2), merge-on-read support, and incremental read APIs are steps toward native streaming support. The Iceberg V3 specification advances this with streaming-native compaction, lineage, and governance primitives that further reduce the impedance mismatch with streaming writers.

Flink and Iceberg integration is tightening. Every Flink release improves the Iceberg sink: better commit coordination, more efficient file creation, improved support for CDC patterns. The gap between Flink-native writes and purpose-built streaming formats is narrowing.

Paimon and Fluss are proving the model. Even if Paimon and Fluss do not become the dominant formats, they are demonstrating that streaming-first table design is viable and that the maintenance burden of streaming writes can be absorbed by the storage system rather than imposed on the operator.

Autonomous maintenance is becoming table stakes. The operational reality of streaming Iceberg tables makes autonomous maintenance — compaction, expiration, cleanup — a requirement, not a luxury. Tools like LakeOps that provide this autonomously are becoming as essential as the ingestion pipeline itself.

The endgame is a lakehouse where the distinction between "streaming data" and "batch data" is irrelevant. Data arrives continuously, is immediately queryable, and the storage layer manages its own health. We are not there yet — but the architecture is converging, the tools are maturing, and the gap between vision and production reality is closing.

Summary

The duplication between Kafka and Iceberg is a structural problem that the streaming lakehouse aims to solve. The table — not the topic, not the file — becomes the unified abstraction for both real-time and analytical workloads.

Today, you can build streaming pipelines on Iceberg with Flink, Kafka Connect, or Spark. Each works. Each creates operational debt in the form of small files, snapshot accumulation, and metadata growth. Managing that debt requires continuous, automated maintenance — not manual scripts or weekly cron jobs.

Projects like Apache Fluss and Apache Paimon are pushing toward a future where the storage system manages its own maintenance. In the meantime, tools like LakeOps provide the autonomous maintenance layer that makes streaming Iceberg tables viable in production — handling compaction, snapshot expiration, and table health monitoring so your team can focus on the data, not the plumbing.

The streaming lakehouse is not a future architecture. It is the current trajectory. Start building for it now — with the tooling that exists today — and the migration to a fully unified streaming lakehouse will be incremental, not revolutionary.