Apache Iceberg V3 for Streaming: Row-Level Lineage, Schema Evolution, and Governance

Apache Iceberg V3 is the most significant specification update since V2 introduced row-level deletes and sequence numbers. For batch analytics teams, V3 is a welcome set of efficiency gains — deletion vectors are faster than positional deletes, default column values eliminate backfills, and row lineage simplifies CDC. But for streaming data platform teams, V3 is something more fundamental: it is the first version of the Iceberg specification that takes streaming governance seriously.

Governance on batch tables is relatively straightforward. You run a pipeline, it finishes, you validate the output, you update lineage metadata. Schemas change on release boundaries. Access control policies are reviewed quarterly. The cadence of change is slow enough that humans can stay in the loop. Streaming breaks all of these assumptions. Data arrives continuously. Schemas evolve without warning — a field appears in Kafka, propagates through Flink, and lands in Iceberg before anyone reviews it. Lineage is not a static DAG; it is a continuously evolving graph of which records, from which sources, through which transformations, at which point in time, produced which output rows. And governance policies must apply in real-time, not after the fact.

V3 does not solve all of these problems. But it provides the specification-level primitives that make solutions possible. Row-level lineage gives every row a persistent identity that survives compaction and rewriting. Default column values allow schema evolution without stopping pipelines. Deletion vectors enable efficient row-level mutations without the metadata explosion that plagued V2 streaming tables. And the REST catalog specification — while not technically part of V3 — provides the governance control plane that ties these features together.

This article explores what Iceberg V3 changes for streaming governance. It covers the technical mechanisms, the integration patterns with REST catalogs, and what data platform teams should plan for as V3 adoption accelerates across streaming engines.

What Iceberg V3 Changes for Streaming Governance

To understand why V3 matters for streaming, you need to understand what V2 lacked. Iceberg V2 was designed primarily with batch workloads in mind. Its governance model assumed periodic commits, stable schemas between commits, and external lineage tracking. Streaming pipelines violated every one of these assumptions:

• Commit frequency. A Flink job writing to Iceberg V2 might commit every 30 seconds. Each commit creates a new snapshot, new manifest files, and potentially new delete files. Over 24 hours, that is 2,880 snapshots. Over a week, 20,160. The metadata layer becomes a governance liability — every lineage query, every access audit, every schema check has to traverse this metadata explosion.
• Schema stability. Batch pipelines evolve schemas on deployment boundaries. Streaming pipelines receive schema changes in real-time from upstream sources — a new field appears in a Kafka topic, a CDC connector picks up a column addition from PostgreSQL, an Avro schema in the Schema Registry gains a new union variant. V2 schema evolution required stopping the pipeline, altering the table, and restarting.
• Lineage granularity. V2 lineage was snapshot-level at best. You could determine that a snapshot was produced by a particular commit, but you could not track which specific rows within that snapshot came from which upstream records. For compliance (GDPR right-to-erasure, for example), this meant you could not answer the question "which output rows were derived from this specific input record?" without scanning entire snapshots.
• Mutation efficiency. Streaming tables with frequent updates (CDC from OLTP databases) required positional delete files in V2. Each update cycle created new delete files that fragmented across partitions, creating thousands of small metadata files that degraded both query performance and governance query performance alike.

V3 addresses each of these pain points with specific specification-level features. It does not eliminate the need for operational maintenance — the physical layer still accumulates debt — but it provides the semantic foundation for governance that V2 lacked.

Streaming tables on V3 commit every 30 seconds, producing thousands of snapshots, manifests, and small files per day. V3's governance primitives — row lineage queries, deletion vector audits, schema metadata lookups — all execute against this physical layer. If the physical layer is unhealthy, governance queries are slow or broken. LakeOps provides autonomous maintenance for high-frequency streaming tables, adapting compaction, snapshot expiration, and orphan cleanup to each table's commit rate and mutation pattern — keeping the physical layer healthy so V3's governance metadata remains queryable at interactive speeds.

Row-Level Lineage: Tracking Records Through Continuous Transformations

Row-level lineage is the headline feature of V3 for governance. The specification mandates that every V3 table must track two hidden metadata fields for every row:

• `_row_id` — a unique long identifier assigned to each row when it is first written to the table. This ID is permanent: it survives compaction, file rewrites, and metadata updates. The same row always has the same _row_id, regardless of which physical file contains it.
• `_last_updated_sequence_number` — the Iceberg sequence number of the commit that last modified the row. This maps directly to a specific snapshot, which in turn maps to a timestamp, a job, and a set of manifest changes.

These fields are not optional. V3 mandates their tracking for all newly created rows. Engines must maintain the next-row-id table field and assign IDs via inheritance — when a row is first added, its _row_id is set to null in the data file and assigned at read time from the file's first_row_id range. This inheritance model means writers do not need to coordinate synchronously to assign unique IDs, which is critical for high-throughput streaming writes where multiple Flink subtasks commit concurrently.

Why Row-Level Lineage Changes the Streaming Governance Story

Before V3, tracking row-level lineage in a streaming context required external infrastructure — a separate lineage store, custom metadata columns baked into your data model, or engine-specific extensions that were not portable across query engines. Each approach had fundamental limitations:

• Custom metadata columns bloated the data model and required every producer to populate them correctly. A Flink job, a Spark job, and a CDC connector would each need custom logic to maintain lineage fields — and a single producer that skipped them would break the chain.
• External lineage stores (Atlas, DataHub, OpenLineage) tracked lineage at the job or dataset level, not the row level. They could tell you that "Job X wrote to Table Y at time T" but not which specific rows were produced or which input records produced them.
• Engine-specific extensions (like Spark's internal row-tracking) were not readable by other engines. Lineage captured by Spark was invisible to Trino queries or Dremio scans.

V3 row lineage is format-level. It is written into the Parquet files themselves. Any engine that reads Iceberg V3 — Spark, Flink, Trino, Dremio, Snowflake — can query lineage columns without any additional configuration. This portability is what makes it viable for multi-engine streaming platforms where data flows through Flink for ingestion, Spark for transformation, and Trino for serving.

Continuous Lineage in Streaming Pipelines

Consider a streaming CDC pipeline: PostgreSQL → Debezium → Kafka → Flink → Iceberg. With V3 row lineage, every row that lands in the Iceberg table carries a permanent _row_id and a _last_updated_sequence_number that tracks when it was last modified. When a row is updated (a customer changes their address), the _row_id remains the same — the row's identity persists — but the _last_updated_sequence_number advances to the new commit's sequence number.

This enables several governance capabilities that were previously impractical on streaming tables:

• Incremental change detection without full-table scans. Downstream consumers can query _last_updated_sequence_number > X to find all rows modified since their last checkpoint. This replaces the expensive snapshot-diff operations that V2 CDC pipelines relied on.
• Right-to-erasure compliance. When a GDPR deletion request arrives, you can identify all rows derived from a specific source record by tracing _row_id values through the transformation chain — provided your transformation layer preserves or maps row IDs.
• Audit trail without external infrastructure. The combination of _row_id and _last_updated_sequence_number, joined against the snapshot metadata table, produces a complete audit trail: which row, modified when, by which commit, as part of which snapshot.
• Streaming deduplication. If a Flink job replays from an earlier Kafka offset (after a failure), row IDs provide a mechanism to detect and resolve duplicates at the table level rather than requiring exactly-once semantics at the transport layer.

Row Lineage Assignment Mechanics

The specification uses an inheritance model for row ID assignment that is designed for concurrent writers — exactly the pattern streaming engines use. Each snapshot maintains a first-row-id field. When a writer creates a new data file, it does not assign row IDs directly into the Parquet columns. Instead, the row ID column is written as null. At read time, the engine assigns row IDs by combining the file's first_row_id (inherited from the manifest) with the row's position within the file.

This approach has important implications for streaming:

• No coordination required between concurrent writers. Multiple Flink subtasks can write data files simultaneously without a distributed sequence counter. Row ID ranges are assigned at commit time, not write time.
• Compaction preserves lineage. When a compaction job rewrites small files into larger ones, it must preserve the original _row_id values. The row's identity is tied to its logical meaning, not its physical location.
• Optimistic concurrency works. Iceberg's optimistic commit model — where writers prepare their commits independently and resolve conflicts at commit time — is fully compatible with row lineage because ID assignment happens during the commit, not during the write.

One important caveat: row lineage does not track lineage for rows updated via equality deletes. Engines using equality deletes avoid reading existing data before writing changes and cannot provide the original row ID for new rows. These updates are treated as if the existing row was completely removed and a new row was added. For streaming pipelines that rely heavily on equality deletes (some CDC patterns), this means lineage continuity is broken at those mutation points. Deletion vectors, which V3 also introduces, provide a path around this limitation.

Schema Evolution for Streaming: Schemas Change Without Warning

In batch analytics, schema evolution is an event. Someone submits a pull request to add a column. It gets reviewed. The migration runs. Downstream consumers are notified. In streaming, schema evolution is a continuous process that happens without ceremony:

• A developer adds a field to a protobuf message. The next deployment starts emitting events with the new field. The Kafka topic's Schema Registry accepts the new schema (if it is backward-compatible). The Flink job consuming that topic now receives records with an extra field.
• A DBA adds a column to a PostgreSQL table. The CDC connector (Debezium) detects the DDL change and emits a schema-change event. The downstream pipeline must handle records with the new column — even though the Iceberg table does not have it yet.
• An upstream team renames a field in their Avro schema. The Schema Registry allows it (if compatibility mode permits). Every consumer of that topic must now handle the rename — or fail.

V2 schema evolution required explicit DDL operations on the Iceberg table before the new schema could be written. For streaming pipelines, this created a deadlock: the pipeline could not write the new schema until someone altered the table, but no one altered the table until they noticed the pipeline was failing. The result was either pipeline downtime or silent data loss (dropping new fields).

V3 Default Values: Schema Evolution Without Pipeline Downtime

V3 introduces default column values as a specification-level feature, and this is the key enabler for streaming schema evolution. When a column is added to a V3 table with a default value, no data files are rewritten. The operation is purely metadata. Older data files that lack the column are read as if the column exists with the default value — the engine applies the default at read time from the schema metadata.

The specification distinguishes between two types of defaults:

• `initial-default` — the value returned for rows in data files that were written before the column existed. This is set once when the column is added and cannot be changed.
• `write-default` — the value used for new rows when the writer does not provide an explicit value. This can be updated over time via schema evolution.

This distinction is critical for streaming pipelines where a column's semantics may evolve. Consider adding a consent_status column to a customer table fed by CDC. The initial-default might be 'unknown' (because historical records predate your consent tracking). The write-default starts as 'unknown' but is later updated to 'pending' when your application starts collecting consent signals. Old data reads as 'unknown'. New records without explicit consent land as 'pending'. The schema evolution is non-disruptive — no pipeline restart required.

Practical Patterns for Streaming Schema Evolution

With V3 default values, streaming teams can implement several patterns that were previously painful:

Pattern 1: Auto-evolution from Schema Registry. The Flink Iceberg sink connector (via FLINK-39055) now supports propagating default values from CDC schemas into Iceberg table metadata. When a new column appears in the upstream source, the connector can automatically issue an ALTER TABLE ADD COLUMN ... DEFAULT operation on the Iceberg table, then continue writing without interruption. The default value is parsed from the CDC schema's default expression into an Iceberg literal.

Pattern 2: Forward-compatible schemas. For Kafka consumers where the full schema is not known in advance (protobuf oneof fields, Avro union types), V3's Variant data type provides an escape hatch. Instead of defining every possible column upfront, streaming pipelines can land semi-structured data into a Variant column and promote specific fields to typed columns when the schema stabilizes. This is particularly useful for event sourcing architectures where event types proliferate faster than schema management can keep up.

Pattern 3: Required columns without backfills. V3 allows adding NOT NULL columns to existing tables — as long as a non-null default is provided. The default satisfies the constraint for historical rows. New rows must provide a value or use the write-default. For streaming pipelines enforcing data quality contracts, this means you can tighten constraints without a full table rewrite and without stopping the pipeline.

Pattern 4: Graceful field deprecation. When a field is removed from the upstream schema (the Kafka producer stops sending it), V3's write-default ensures that new rows still satisfy any downstream expectations of the field's presence. The column can be marked for deletion in a future schema evolution step, but existing pipelines and queries continue to work during the transition period.

Deletion Vectors: Efficient Mutations on Streaming Tables

V2's approach to row-level deletes — positional delete files — was designed for batch workloads where deletes are infrequent and batched. For streaming tables receiving continuous CDC updates, positional deletes were a disaster. Every update cycle created new delete files. Over hours of streaming, a single partition could accumulate hundreds of small delete files. Query engines had to merge these files at read time, and the merge cost grew linearly with the number of delete files.

V3 replaces positional deletes with binary deletion vectors — compact bitmaps stored alongside data files that mark which rows are logically deleted. The difference for streaming workloads is dramatic:

• Space efficiency. A deletion vector for a data file is a single bitmap, not a separate file per delete operation. Multiple updates to the same data file produce a single, compacted deletion vector rather than multiple delete files.
• Read performance. Engines apply the bitmap at scan time with minimal overhead — it is a single random-access read per data file, not a join across dozens of delete files.
• Compaction simplification. Compaction on V3 tables rewrites data files by applying deletion vectors and producing clean files. The compaction logic is simpler because it does not need to reconcile overlapping delete files.
• Metadata reduction. Streaming tables on V3 produce far fewer metadata objects per unit time. Fewer manifests, fewer files to track, fewer entries for the catalog to manage.

For streaming governance specifically, deletion vectors reduce the metadata noise that makes governance queries expensive. When a compliance team needs to audit which rows were deleted and when, the deletion vector history (tracked via snapshot metadata) provides a cleaner audit trail than the fragmented delete-file approach of V2.

https://www.youtube.com/watch?v=ELJoFADLyeI

Integrating with REST Catalogs for Fine-Grained Streaming Governance

The Iceberg REST catalog specification — while technically independent of V3 — is the governance control plane that makes V3's features actionable for streaming workloads. REST catalogs (Apache Polaris, Unity Catalog, Snowflake Open Catalog, AWS Glue) provide the centralized metadata layer where access control, lineage aggregation, and policy enforcement happen.

For streaming data, the REST catalog integration with V3 features enables several governance patterns:

Scan-Time Policy Enforcement

When a query engine requests a table scan plan from the REST catalog, the catalog can inject access policies — column masking, row filtering, time-based access restrictions — into the scan plan itself. For streaming tables where data freshness varies by partition, this enables time-aware governance: "analysts can see data older than 24 hours; only the pipeline service account can see data from the last 24 hours." The REST catalog evaluates the policy at scan-planning time, not at query time, making enforcement efficient regardless of table size.

Credential Vending for Streaming Writers

Streaming jobs run continuously and need storage credentials that refresh without restart. The REST catalog's credential vending APIs provide short-lived credentials (STS tokens on AWS, service account impersonation on GCP) that the streaming job refreshes periodically. This eliminates the operational nightmare of credential rotation on long-running Flink jobs — the catalog handles credential lifecycle, and the streaming job never holds static credentials.

Namespace-Level Governance for Streaming Pipelines

REST catalogs organize tables into namespaces with inheritable policies. For streaming platforms with dozens of tables (one per Kafka topic, one per CDC source), namespace-level governance means you set policies once at the namespace level and they apply to every table created by the streaming pipeline. When a new CDC source is onboarded, the table inherits the namespace's access controls, retention policies, and lineage tracking configuration automatically.

Cross-Engine Lineage Aggregation

When multiple engines write to the same Iceberg table (Flink for ingestion, Spark for transformation, Dremio for serving), the REST catalog provides the single point where lineage from all engines is aggregated. V3's row-level lineage fields are written by each engine independently, but the catalog provides the query interface where governance teams can trace a row's lifecycle across engines — from Flink ingestion through Spark transformation to Dremio serving.

BYOL: Bring Your Own Lineage

The Iceberg ecosystem is converging on a model where catalogs provide the governance framework but lineage content comes from multiple sources — a pattern increasingly referred to as Bring Your Own Lineage (BYOL). This reflects the reality that no single system captures end-to-end lineage in a streaming platform:

• OpenLineage captures job-level lineage from Flink, Spark, and Airflow — which jobs ran, what they read, what they wrote.
• Iceberg V3 row lineage captures row-level lineage within the table — which rows exist, when they were modified, in which commit.
• Schema Registry captures schema lineage — how schemas evolved over time, which versions are compatible, which consumers use which version.
• REST catalog metadata captures table-level lineage — which tables exist in which namespaces, with which access policies, and which engines access them.

BYOL means the catalog exposes APIs for registering external lineage data alongside its native metadata. Unity Catalog's lineage APIs, for example, allow you to register lineage from external systems directly into its governance framework. Apache Polaris's Generic Tables feature lets teams register lineage metadata as governed assets alongside the data tables. The catalog becomes the aggregation point — not the sole source — of lineage.

For streaming teams, the BYOL model is essential because lineage crosses system boundaries constantly. A single streaming record might:

1. 1.Originate in PostgreSQL (captured by Debezium's source connector lineage)
2. 2.Transit through Kafka (tracked by Schema Registry compatibility checks)
3. 3.Transform in Flink (captured by OpenLineage's Flink integration)
4. 4.Land in Iceberg (tracked by V3's _row_id and _last_updated_sequence_number)
5. 5.Get queried by Trino (tracked by the REST catalog's audit log)

No single system sees the full picture. BYOL means the governance platform can assemble the complete lineage graph by pulling from each system's native lineage capabilities. V3's row lineage is one piece — arguably the most important piece for the data-at-rest portion — but it works in concert with job-level, schema-level, and catalog-level lineage to provide end-to-end traceability.

Practical Patterns for Streaming Sources

Kafka to Iceberg V3 with Schema Evolution

The most common streaming-to-Iceberg pattern is Kafka → Flink → Iceberg. With V3, this pipeline gains native support for schema evolution and row lineage without custom code:

Step 1: Schema Registry as source of truth. Kafka topics use Avro or Protobuf schemas registered in Confluent Schema Registry (or AWS Glue Schema Registry). Schemas evolve with backward compatibility — new fields are added with defaults, removed fields have defaults applied.

Step 2: Flink consumes with schema awareness. The Flink Kafka consumer deserializes records using the latest compatible schema. When a new field appears, Flink's type system accommodates it (either via schema evolution in the Flink job, or via a Variant/MAP column that captures unknown fields).

Step 3: V3 Iceberg sink auto-evolves. The Flink Iceberg sink connector detects schema mismatches and issues ALTER TABLE ADD COLUMN operations with appropriate defaults. The Iceberg table evolves without pipeline restart. Row lineage metadata is written automatically — every row gets a _row_id and _last_updated_sequence_number.

Step 4: REST catalog governs. The REST catalog (Polaris, Unity, or cloud-native) provides access control, audit logging, and lineage aggregation. Credential vending keeps the Flink job's S3/GCS access current without restart.

CDC Sources with V3 Deletion Vectors

CDC from OLTP databases (PostgreSQL, MySQL, MongoDB) produces a mix of inserts, updates, and deletes. On V2, this pattern was operationally painful — every update generated a positional delete file plus a new data file. On V3:

Inserts are written normally with row lineage metadata. Each new row gets a unique _row_id.

Updates are handled via deletion vectors plus new row writes. The old row is marked deleted in the deletion vector (a bitmap update, not a new file). The new row is written with the same logical identity — though note that if the engine uses equality deletes rather than position-based updates, the _row_id will be different (a limitation of the V3 spec).

Deletes are pure deletion vector updates. The row is marked in the bitmap. For compliance auditing, the deletion vector history (accessible via snapshot metadata) records when the deletion happened.

This pattern produces far fewer metadata objects than V2's approach, which means governance queries ("show me all mutations to this row in the last 30 days") execute in seconds rather than minutes.

Multi-Source Streaming with Unified Governance

Production streaming platforms rarely have a single source. A typical e-commerce platform might have:

• Clickstream events from Kafka (high volume, append-only)
• Order events from Kafka (medium volume, keyed)
• Customer CDC from PostgreSQL via Debezium (low volume, high mutation)
• Product catalog CDC from MongoDB (low volume, complex schemas)
• Session data from Redis CDC (medium volume, TTL-based expiry)

Each source has different schema evolution patterns, different mutation rates, and different governance requirements. V3 provides a unified framework:

• All tables get row lineage automatically — no source-specific configuration.
• Schema evolution uses the same mechanism (default values) regardless of source.
• Deletion vectors handle mutations uniformly whether the source is PostgreSQL CDC or Kafka tombstones.
• The REST catalog applies namespace-level policies across all tables from all sources.

The governance team does not need to know which source a table comes from to apply consistent policies. The V3 specification normalizes the governance interface across heterogeneous streaming sources.

What This Means for Data Platform Teams

If you are building a streaming platform on Iceberg, V3 changes your governance strategy in several concrete ways:

Plan for V3 Adoption Now, Even If Engine Support Is Partial

As of mid-2026, V3 support varies by engine. Apache Spark (with Iceberg 1.11.0+) has full V3 read and write support including row lineage. Apache Flink supports deletion vectors and DV compaction fully; row lineage support is in progress. Trino supports deletion vectors for reads and writes but row lineage is not yet fully available. Dremio has comprehensive V3 support including row lineage in query planning and result delivery. Snowflake supports V3 natively with row lineage GA since May 2026.

The upgrade path is metadata-only and non-destructive. Existing V2 data files remain valid. But if any engine in your pipeline does not support V3 writes, hold off — a V3 table that receives incorrect writes from an incompatible engine will have data consistency problems. The practical advice: upgrade tables where all writers support V3, and keep tables with mixed-engine writers on V2 until the lagging engine catches up.

Row Lineage Reduces But Does Not Eliminate External Lineage Infrastructure

V3 row lineage tells you which rows exist and when they were last modified. It does not tell you which upstream source record produced them, which transformation logic was applied, or which job executed the write. You still need OpenLineage (or equivalent) for job-level lineage, Schema Registry for schema lineage, and catalog audit logs for access lineage. What V3 eliminates is the need for custom, per-table row-tracking columns — the specification handles that for you, portably across engines.

Deletion Vectors Change Your Compaction Strategy

On V2, compaction for streaming tables was primarily about merging small files. On V3, compaction also means applying deletion vectors — rewriting data files with their deletion vectors applied to produce clean files without logical deletions. This changes when and how aggressively you compact. Tables with high mutation rates (CDC sources) benefit from more frequent compaction to apply accumulated deletion vectors. Tables with low mutation rates (append-only event streams) compact primarily for file size, as they always have. LakeOps adapts compaction triggers based on both file size and deletion vector density, ensuring streaming tables stay query-performant regardless of mutation rate. See also our guides on Iceberg schema evolution and streaming lakehouse patterns.

The REST Catalog Is Your Governance Boundary

For streaming platforms, the REST catalog is not optional infrastructure — it is the governance boundary. Every engine in your pipeline should access Iceberg tables through the REST catalog, not through direct metadata file access. This ensures that access policies, credential vending, and audit logging are applied consistently regardless of which engine is reading or writing. Apache Polaris (now a top-level ASF project) is the vendor-neutral standard; Unity Catalog and Snowflake Open Catalog are alternatives with deeper integration into their respective ecosystems. See our managed Iceberg solutions for guidance on catalog-integrated table management.

Governance Is Only as Good as Table Health

V3's governance features — row lineage queries, deletion vector auditing, schema metadata lookups — all execute against the table's physical layer. If that layer is unhealthy (thousands of small files, hundreds of expired snapshots, orphaned data files consuming storage), governance queries are slow, expensive, or outright broken. A lineage audit that takes 45 minutes because it scans 12,000 tiny files is not a functioning governance capability. LakeOps continuously maintains the physical layer — compacting files, expiring snapshots, cleaning orphans — so that V3's governance metadata remains queryable at interactive speeds. This is not optimization for its own sake; it is a prerequisite for governance to function.

The Governance-Ready Streaming Lakehouse

Iceberg V3 represents a philosophical shift in how the specification views streaming workloads. V1 and V2 treated streaming as a special case — something that could be bolted on with connectors and maintenance scripts. V3 treats streaming as a first-class citizen with specification-level features designed for continuous, high-frequency, multi-engine workloads.

Row-level lineage means every row has a persistent identity. Default column values mean schemas evolve without disruption. Deletion vectors mean mutations are efficient. REST catalogs mean governance policies are enforceable across engines. Together, these features form a coherent governance framework for streaming data that did not exist at the specification level before.

But specification features are only the foundation. The governance-ready streaming lakehouse requires operational discipline: continuous table maintenance to keep the physical layer healthy, monitoring to detect when governance queries degrade, and automation to enforce policies without human intervention. V3 provides the primitives. Building the governance-ready streaming lakehouse on top of them is the work of data platform teams — and the platforms that support them.