Apache Iceberg lakehouse governance is not embedded in the table format — and that is a deliberate design decision. Governance in an open lakehouse must be decomposed into three distinct layers: the table format (data portability), the catalog control plane (enforcement), and the policy engine (rules). Conflating any two of these layers produces a system that is either too rigid to evolve or too porous to enforce.
This separation matters because production lakehouses are not single-engine, single-catalog environments. They are multi-engine (Spark, Trino, Flink, StarRocks, DuckDB), multi-catalog (Polaris, Nessie, Glue, Gravitino), and increasingly multi-cloud. Governance that is hardcoded into the format, baked into a single engine, or locked to a single catalog cannot survive this reality. The governance model must be as modular and interoperable as the storage format it protects.
Governance defines who can access data — which identities can read which columns of which tables under which conditions. But access control alone does not guarantee that data is actually usable. Operations ensure the data is accessible — fast, healthy, and cost-efficient. A table that passes every governance check but takes 45 minutes to query because of 200,000 small files is technically governed but operationally useless. Understanding how governance and operations interact — and why each needs its own control plane — is a recurring theme in this article.
The governance layer manages policies and access. The operational layer — filled by LakeOps as a control plane for Apache Iceberg lakehouses — handles autonomous table maintenance, health monitoring, and compaction across all engines. Together they ensure that governed data is also structurally healthy, query-performant, and cost-efficient.
This guide covers why Iceberg intentionally avoids embedding governance in metadata, how the three-layer model works in practice, how Apache Polaris serves as the enforcement point, how pluggable policy engines integrate with the catalog, and why this architecture is essential for multi-engine environments.
Why governance must stay separate from the table format
The temptation to embed governance directly into the table format is understandable. If Iceberg metadata included access control lists, column masking rules, and row-level security policies, every engine reading the table would automatically enforce those rules. No separate catalog needed, no external policy engine — just read the metadata and apply the policies.
The Iceberg community rejected this approach for three fundamental reasons.
Portability would be destroyed. Iceberg's defining value proposition is that any engine can read any Iceberg table through a standard metadata specification. The moment governance rules are embedded in metadata, every engine must understand and implement the same governance model — the same RBAC structure, the same masking functions, the same row-level filter syntax. Any engine that does not implement the full governance model either breaks compatibility or silently bypasses security. This is exactly the kind of vendor-specific coupling that Iceberg was designed to eliminate.
Evolution would be frozen. Governance requirements change faster than storage formats. Regulatory regimes evolve (GDPR, CCPA, DORA, AI Act). Organizational structures change (teams merge, roles are redefined, data domains are reclassified). Classification taxonomies expand (PII subclasses, data residency labels, retention categories). If governance rules are encoded in Iceberg metadata, every change to a governance policy requires a metadata migration across every affected table — potentially millions of metadata files across thousands of tables. Governance policies need to change at the speed of policy, not at the speed of metadata migration.
Enforcement would be unverifiable. When governance rules live in the format, enforcement depends on the reader. A well-behaved Spark job reads the ACL and respects it. A misconfigured Trino deployment ignores it. A custom Parquet reader bypasses it entirely. There is no enforcement authority — only conventions that cooperative engines follow and adversarial engines ignore. Centralized enforcement through a catalog, by contrast, means that no engine can read the table without first passing through the enforcement point. The catalog is the chokepoint, and chokepoints are where security is enforced.
Iceberg's position is clear: the table format is responsible for data portability — organizing data files, manifests, snapshots, and metadata in a way that any engine can read efficiently. Governance is someone else's job. The question is whose job, and how the responsibilities decompose.
Three layers: format, catalog, and policy engine
The governance architecture that the ecosystem is converging on consists of three layers, each with a distinct responsibility.
Layer 1: Table format — data portability and structural integrity
Iceberg defines how data is physically organized: immutable Parquet data files, manifest files that index them, manifest lists that compose snapshots, and metadata files that track the table's schema, partition spec, sort order, and snapshot history. The format guarantees ACID transactions, schema evolution, partition evolution, and time travel — all without any governance semantics.
What the format does carry is structural metadata that governance systems can leverage: column names and types (for classification), partition specs (for partition-level access control), snapshot history (for audit trails), and table properties (for custom labels that governance systems interpret). But the format never interprets these — it stores them as data, not as rules.
This is the correct boundary. The format says what the data looks like. It does not say who can see it.
Layer 2: Catalog control plane — enforcement and coordination
The catalog is the enforcement layer. In the REST Catalog protocol, every operation — table creation, schema alteration, data read, snapshot commit — routes through the catalog. This makes the catalog the natural point for access control enforcement because it is the mandatory intermediary between engines and data.
Apache Polaris, as a REST-first catalog, enforces governance at this layer. When a Spark job requests access to a table, Polaris evaluates the request against its access control model before returning the metadata location. If the identity lacks permission, the request is denied before any data is read. This is fundamentally different from format-level governance — the engine never sees the metadata, let alone the data, unless the catalog authorizes it.
The catalog also serves as the coordination point for governance-relevant operations. Table creation triggers classification workflows. Schema changes trigger re-evaluation of masking rules. Branch and tag management enables governance over different versions of the data. Multi-table transactions (covered in our multi-table transactions guide) require coordinated governance checks across all participating tables.
The catalog's role extends beyond access control to operational coordination. It is the control plane — not just for metadata, but for the lifecycle of every table. Who can create tables in a namespace? Who can alter schemas? Who can drop tables? Who can run compaction? These are governance questions that the catalog answers through its access control model.
Layer 3: Policy engine — rules, classification, and context
The policy engine defines the rules that the catalog enforces. This separation is critical: the catalog is the enforcement mechanism, but it should not be the rule authoring system. Policy engines like Open Policy Agent (OPA), Apache Ranger, and cloud-native IAM systems are purpose-built for defining, versioning, auditing, and distributing access policies.
Policy engines bring capabilities that catalogs should not replicate: rule versioning and rollback, policy simulation (what-if analysis before deploying a new rule), cross-system policy consistency (the same policy engine governs databases, object storage, and lakehouse tables), audit trails for policy changes (who changed what rule, when, and why), and attribute-based access control (ABAC) that evaluates context beyond identity — time of day, source IP, query intent, data classification level.
The three-layer model means: the policy engine defines the rules, the catalog enforces them, and the table format is unaware of both. Each layer can evolve independently. You can swap Ranger for OPA without changing the catalog. You can migrate from Glue to Polaris without rewriting your policies. You can upgrade from Iceberg v2 to v3 without touching your governance model.
How Apache Polaris serves as the enforcement point
Apache Polaris (incubating) is designed from the ground up as a governance-aware Iceberg catalog. Its architecture reflects the three-layer model — Polaris is the catalog enforcement layer that integrates with external policy engines for rule definition.
Namespace-level and table-level access control
Polaris implements a hierarchical access control model. At the top level, catalogs contain namespaces, and namespaces contain tables. Access grants cascade: a principal granted CATALOG_MANAGE_CONTENT on a catalog inherits management rights over all namespaces and tables within it. A principal granted NAMESPACE_MANAGE_CONTENT on a namespace inherits rights over all tables in that namespace.
The privilege model includes granular operations: TABLE_READ_DATA, TABLE_WRITE_DATA, TABLE_CREATE, TABLE_DROP, TABLE_LIST, NAMESPACE_CREATE, NAMESPACE_LIST, and administrative privileges for managing grants themselves. This granularity enables the principle of least privilege — a data scientist can read tables in the analytics namespace without being able to create tables, alter schemas, or access tables in the raw namespace.
Service principals and credential vending
Polaris separates identity from access through service principals. Engines authenticate to Polaris with OAuth2 credentials, receiving short-lived tokens scoped to specific operations. Polaris then vends storage credentials (S3 temporary credentials, GCS signed URLs, Azure SAS tokens) that are scoped to exactly the data files the engine is authorized to access.
This is credential vending — the foundation of zero-trust lakehouse security — and it solves one of the hardest problems in lakehouse governance. Without credential vending, engines need direct access to the storage bucket, which means they can read any file in the bucket regardless of table-level access controls. With credential vending, the engine receives credentials that only allow access to the specific data files in the specific table it is authorized to read. The catalog is the bottleneck through which all storage access flows.
Catalog-enforced operations
Every REST Catalog operation passes through Polaris's authorization layer. This includes not just data reads and writes, but also administrative operations that have governance implications:
Schema evolution — Adding, dropping, or renaming columns changes the structure of governed data. A column rename that changes ssn to customer_id might inadvertently bypass a masking rule that targets ssn by name. Polaris can require additional authorization for schema changes that affect classified columns.
Snapshot management — Expiring snapshots removes time-travel capability and can permanently delete data. In regulated environments, snapshot expiration must be governed — certain tables may require minimum retention periods for compliance. Polaris can enforce retention policies by rejecting snapshot expiration operations that would violate them.
Table drops — Dropping a table is irreversible in most deployments. Polaris can enforce soft-delete policies (tables are marked for deletion but not immediately purged), require additional approval workflows for tables in specific namespaces, or restrict drops to specific administrative roles.
Pluggable policy interfaces — integrating with existing policy systems
Production enterprises do not deploy governance from scratch. They already have policy systems — OPA for microservices authorization, Ranger for Hadoop-era data access, cloud IAM for infrastructure permissions, and custom LDAP/AD integrations for identity management. A governance architecture that requires replacing all of these is dead on arrival.
The three-layer model solves this through pluggable policy interfaces. The catalog exposes a policy evaluation API — when an access request arrives, the catalog calls out to the configured policy engine with the request context (identity, operation, resource, attributes) and receives an allow/deny decision. The catalog enforces the decision without needing to understand the policy logic.
Open Policy Agent (OPA)
OPA is the emerging standard for policy-as-code in cloud-native architectures. Policies are written in Rego, a declarative language that evaluates structured input against rules. In the lakehouse governance context, OPA excels at attribute-based access control (ABAC) — policies that consider not just who is requesting access, but the full context of the request.
An OPA policy for Iceberg table access might evaluate: the requesting principal's team membership and clearance level, the table's data classification (PII, financial, public), the columns being accessed (allowing access to non-sensitive columns while denying access to PII columns), the time window (restricting access to production data outside business hours), and the query pattern (allowing aggregations but denying row-level exports of sensitive data).
OPA policies are versioned in Git, tested with unit tests, and deployed through CI/CD pipelines — the same workflows that manage application code. This is governance-as-code: auditable, reviewable, and reproducible.
Apache Ranger
Ranger has been the governance standard in Hadoop ecosystems for over a decade. Organizations with existing Ranger deployments — policies, audit logs, classification taxonomies — need a migration path that preserves their investment. Polaris's pluggable policy interface allows Ranger to continue serving as the policy engine while Polaris serves as the enforcement catalog.
Ranger's strengths are its mature UI for policy management, its tag-based policies that work with Apache Atlas classifications, and its comprehensive audit logging. The integration pattern is straightforward: Polaris delegates authorization decisions to Ranger's REST API, passing the table identity, namespace, operation type, and requesting principal. Ranger evaluates the request against its policy store and returns the decision. Polaris enforces it.
This integration means that teams migrating from Hive Metastore to Polaris can retain their existing Ranger policies — the enforcement point changes, but the rules remain the same. For catalog migration strategies, see our catalog migration guide.
Cloud-native IAM integration
AWS IAM, Google Cloud IAM, and Azure RBAC are the default authorization systems for cloud infrastructure. In cloud-native lakehouses, integrating these with catalog-level governance provides a unified permission model — the same IAM role that grants access to a Kubernetes namespace also grants access to specific Iceberg namespaces.
The integration pattern varies by cloud. On AWS, Polaris can validate IAM role trust policies and map IAM roles to catalog principals. On GCP, Polaris can accept Google-signed identity tokens and evaluate access against Workload Identity Federation. On Azure, Polaris can integrate with Entra ID (Azure AD) for identity and map security groups to catalog grants.
The key principle is that the catalog does not replace cloud IAM — it layers table-level governance on top of it. Cloud IAM controls who can reach the catalog. The catalog controls who can access specific tables. The policy engine defines the rules for both.
Why this separation matters for multi-engine environments
The three-layer governance model is not just architecturally clean — it is operationally necessary in multi-engine lakehouses. When Spark, Trino, Flink, and StarRocks all access the same Iceberg tables, governance must be engine-independent. The alternative — configuring access control separately in each engine — is a security nightmare.
Engine-level governance does not compose. If Spark enforces access control through its own authorization plugin, Trino through its own system access control, and Flink through its security configuration, you have three independent governance systems that must be kept in sync. Any drift — a permission granted in Spark but not in Trino — creates a bypass path. Any new engine added to the architecture requires implementing the full governance model from scratch. Any policy change must be deployed to every engine independently.
Catalog-level governance composes naturally. When Polaris enforces access control, every engine that connects through the REST Catalog protocol inherits the same governance model. A data scientist who is denied access to the financial.transactions table is denied regardless of whether they query through Spark, Trino, or DuckDB. A new engine added to the architecture inherits the full governance model the moment it connects to Polaris. A policy change propagates to every engine immediately because the enforcement point is singular.
Cross-engine audit becomes possible. With engine-level governance, audit logs are scattered across engine-specific log systems. With catalog-level governance, every access request — regardless of which engine initiated it — is logged in a single audit trail. This is essential for compliance: auditors do not want to correlate Spark logs, Trino logs, and Flink logs to reconstruct who accessed what. They want a single, authoritative record.
For teams operating multi-engine lakehouses, the combination of catalog-level governance (who can access data) and operational coordination from LakeOps (ensuring the data is queryable across all engines) provides end-to-end control without engine-specific configuration. For multi-engine architecture patterns, see our multi-engine architecture guide.
Governance patterns: RBAC, ABAC, and data classification
The three-layer model supports multiple governance patterns, each appropriate for different organizational maturity levels and regulatory requirements.
RBAC at the catalog level
Role-based access control is the foundation. Roles map to organizational functions — data_engineer, data_scientist, analyst, platform_admin — and each role carries a set of catalog-level privileges. This is what Polaris implements natively through its grant model.
Effective RBAC in a lakehouse requires a namespace strategy that aligns with organizational boundaries. A common pattern is three tiers: raw namespaces (restricted to data engineers and ingestion pipelines), curated namespaces (accessible to data scientists and analysts), and published namespaces (broadly accessible for dashboards and reporting). Each tier has different governance requirements — raw may contain unmasked PII, curated has PII masked or tokenized, and published contains only aggregated, non-sensitive data.
The role hierarchy cascades through these tiers: data engineers have write access to raw and curated, data scientists have read access to curated and write access to their team-specific namespaces, and analysts have read access to published. Platform administrators have management access everywhere. This structure is simple, auditable, and covers 80% of governance requirements.
ABAC via policy engines
Attribute-based access control goes beyond roles to evaluate contextual attributes. ABAC is necessary when access decisions depend on the data itself (its classification), the request context (time, location, query type), or dynamic conditions that cannot be captured in static role assignments.
ABAC is implemented through the policy engine layer — OPA, Ranger, or custom policy services. The catalog passes the full request context to the policy engine, which evaluates it against ABAC rules. Examples of ABAC policies in production lakehouses:
Classification-based access: Tables tagged as PII require the requesting principal to have the pii_cleared attribute. Tables tagged as financial require the sox_compliant attribute. The classification is a property of the table (stored in catalog metadata or a classification service), and the attribute is a property of the identity (stored in the identity provider).
Column-level masking: Queries accessing email, phone, or ssn columns return masked values unless the principal has the unmask_pii privilege. The masking is applied at the catalog level through view-based rewriting or at the engine level through UDFs that the catalog mandates.
Time-based restrictions: Access to production data is restricted to business hours for non-service-account identities. This prevents ad-hoc production queries during maintenance windows or outside supervised operating hours.
Purpose limitation: Under GDPR, personal data can only be processed for its stated purpose. ABAC policies can evaluate a purpose tag attached to each query — analytics, ml_training, debugging — and restrict access based on whether the table's data processing agreement covers that purpose.
Data classification as the foundation
Both RBAC and ABAC depend on knowing what the data contains. Data classification — the systematic labeling of tables, columns, and values according to sensitivity, regulatory regime, and business domain — is the foundation that makes governance policies actionable.
Classification operates at multiple granularities. Table-level classification labels entire tables: public, internal, confidential, restricted. Column-level classification labels individual columns: PII, PHI, financial, credential. Value-level classification detects sensitive patterns in the data itself: Social Security numbers, credit card numbers, email addresses.
In the three-layer model, classification metadata can be stored as table properties in Iceberg metadata (e.g., table.classification = confidential), as tags in the catalog (Polaris namespace or table tags), or in a dedicated classification service (Apache Atlas, Collibra, Alation) that the policy engine queries during evaluation. The key is that the classification is referenced by the policy engine and enforced by the catalog — it does not need to be embedded in the table format.
The catalog as control plane — not just metadata
Framing the catalog as merely a metadata store undersells its role. In the three-layer governance model, the catalog is the control plane for the lakehouse — the operational hub through which all table lifecycle operations flow.
Metadata management
The traditional catalog role: storing the mapping from table names to metadata locations, tracking current snapshots, and serving metadata to engines. This is necessary but insufficient for governance.
Access enforcement
The governance role: evaluating every request against the access control model, integrating with policy engines for complex decisions, vending scoped credentials for authorized access, and logging every access for audit.
Operational coordination
The control plane role: coordinating table lifecycle operations across engines. This includes managing concurrent access (multiple engines reading and writing the same tables), sequencing maintenance operations (compaction, snapshot expiration, orphan cleanup), and enforcing operational policies (who can compact, who can expire snapshots, who can run data quality checks).
This third role — operational coordination — is where governance and operations intersect. The catalog knows who has access to a table (governance). It also knows who is actively writing to it, when the last compaction ran, and whether the table is in a healthy state (operations). Combining both gives the catalog the context to make coordinated decisions: do not expire snapshots while a compliance audit is reading historical data; do not compact while a streaming pipeline is actively checkpointing; do not drop a table that still has active downstream consumers.
How governance and maintenance interact
One of the most overlooked aspects of lakehouse governance is its intersection with table maintenance. Compaction, snapshot expiration, orphan file cleanup, and manifest rewriting are not just operational tasks — they are governance-relevant actions that can affect data availability, auditability, and compliance.
Who can compact?
Compaction rewrites data files — it reads existing files, merges them, and writes new, larger files. This requires both read and write access to the table's data. In a governed environment, compaction should be restricted to service accounts with maintenance privileges, not granted to general-purpose data engineering roles.
The governance question is subtle: compaction changes the physical layout without changing the logical data. But a malicious or buggy compaction job could introduce data corruption, alter sort order (affecting downstream query performance), or fail mid-operation leaving the table in a degraded state. Restricting compaction to a trusted maintenance service — and auditing every compaction operation — is a governance requirement, not just an operational preference.
Who can expire snapshots?
Snapshot expiration permanently removes the ability to time-travel to older versions of the data. In regulated environments, this has compliance implications — financial data may need to be queryable at any historical point for seven years, healthcare data may need to be auditable for the lifetime of the patient relationship.
Governance policies for snapshot expiration should include minimum retention periods per classification level (e.g., financial tables retain 7 years, operational tables retain 90 days), approval workflows for expiring snapshots on compliance-sensitive tables, and audit logging that records which snapshots were expired, by whom, and when. The catalog enforces these policies by rejecting expiration requests that violate retention rules.
Who can delete orphan files?
Orphan files — data files that exist in storage but are not referenced by any snapshot — are created by failed writes, aborted compaction jobs, and metadata inconsistencies. Cleaning them up reclaims storage and reduces cost. But aggressive orphan cleanup can delete files that are still needed by in-progress operations or by external systems that reference them directly.
Governance for orphan cleanup requires a grace period (never delete files younger than a threshold — typically 3–7 days), validation against all active snapshots and in-progress transactions, and restriction to service accounts that understand the table's operational state. A general-purpose data engineer should not be running ad-hoc orphan cleanup on production tables. For orphan cleanup patterns and risks, see our orphan files cleanup guide.
The maintenance-governance coordination gap
Most governance systems handle read and write access well. Few handle maintenance operations well. The result is a gap: the governance system controls who can query and who can ingest, but maintenance operations — compaction, expiration, cleanup, manifest rewriting — operate outside the governance model, often running as overprivileged service accounts with blanket access.
LakeOps fills this gap by serving as the dedicated maintenance authority. Instead of granting compaction privileges to multiple engines and hoping they coordinate, LakeOps runs all maintenance operations through a single control plane that respects table-level policies, avoids conflicts with active writers, sequences operations correctly, and logs every action. The governance catalog controls who can read and write data. LakeOps controls who (and when and how) maintains it. For the full operational model, see the managed Iceberg solution.
Governance in practice: building the three-layer stack
For teams designing governance for a production Iceberg lakehouse, the three-layer model suggests a clear implementation path.
Step 1: Choose a governance-capable catalog
The catalog is the enforcement point. If it does not support fine-grained access control, credential vending, and policy engine integration, no amount of policy sophistication upstream will matter. Apache Polaris, Project Nessie, and Apache Gravitino (incubating) all support REST Catalog protocol with varying degrees of governance capability. AWS Glue provides basic IAM-based access control but lacks the fine-grained privilege model of purpose-built governance catalogs. For a detailed comparison, see our catalog comparison guide.
Step 2: Establish a namespace and classification strategy
Before writing policies, define the organizational structure. Map namespaces to data domains or sensitivity tiers. Classify tables and columns by sensitivity level. This classification is the input to every governance policy — without it, policies have nothing to evaluate against.
A practical starting point: three sensitivity tiers (public, internal, restricted), one namespace per data domain per tier (e.g., internal.marketing, restricted.finance), and column-level classification for PII fields. This structure is simple enough to implement in a week and comprehensive enough to cover most governance requirements.
Step 3: Implement RBAC as the baseline
Start with role-based access control using the catalog's native privilege model. Define roles that map to organizational functions, grant namespace-level and table-level privileges per role, and enforce through the catalog. This covers the majority of access control requirements and is auditable, understandable, and manageable without external tooling.
Step 4: Layer ABAC through a policy engine
Once RBAC is in place, add attribute-based policies for requirements that roles cannot express: classification-based access, column masking, time restrictions, purpose limitation. Deploy OPA or Ranger as the policy engine, integrate it with the catalog through the pluggable policy interface, and version policies in Git alongside application code.
Step 5: Close the operational loop
Governance without operational health is incomplete. A governed table that is structurally degraded — millions of small files, thousands of expired snapshots, orphaned data consuming storage — passes every access control check but fails every performance expectation. Close the loop by connecting the governance catalog (who can access) with an operational control plane like LakeOps (ensuring the data is healthy, fast, and cost-efficient). The combination provides end-to-end lakehouse management: governed access to well-maintained data.
The convergence of governance and operations
The open lakehouse disaggregated storage, compute, and metadata into independent layers. Governance must follow the same decomposition — rules separate from enforcement, enforcement separate from the format. The three-layer model (format → catalog → policy engine) is the architectural pattern that makes this decomposition work in practice.
Apache Polaris provides the enforcement layer — credential vending, fine-grained privilege management, and policy engine integration through a REST-first catalog. OPA, Ranger, and cloud IAM systems provide the policy layer — rule definition, attribute-based evaluation, and audit trails. Iceberg provides the format layer — portable, open, and deliberately governance-unaware.
But governance is only half the picture. The other half is operations. Governance controls who can access data. Operations ensure the data is worth accessing — structurally healthy, query-performant, and cost-efficient. The catalog serves as the governance control plane. LakeOps serves as the operational control plane. Together, they provide the two pillars that every production lakehouse needs: secure access to healthy data.
For teams building production Iceberg lakehouses: start with the catalog choice (it determines your governance ceiling), layer policies incrementally (RBAC first, ABAC when needed), and close the operational loop from day one. The teams that treat governance and operations as separate concerns — each with its own dedicated control plane — are the ones running multi-engine, multi-cloud lakehouses at scale without governance gaps or operational drift.
For related reading: choosing the best catalog for Apache Iceberg covers the catalog decision in depth, catalog migration covers moving between catalogs while preserving governance, and managed Iceberg shows how LakeOps complements governance catalogs with automated operational health.
