Data LakehouseApache IcebergLakeOps
Data Lakehouse Architecture Guide
A practical guide to building a data lakehouse: object storage, Apache Iceberg, the catalog, the engine fleet, and the control plane that keeps every table fast and cheap.
Back to all articles
Compaction articles
Iceberg compaction strategies, engines, and automation — binpack, sort, Z-order, streaming compaction, and production benchmarks.
48 articles
AIApache IcebergLakeOps
AI Agents for Apache Iceberg Maintenance and Compaction
AI agents reason about table health, decide what maintenance each Iceberg table needs, execute operations in the correct sequence, and learn from outcomes. A deep technical guide covering compaction strategies, the Rust/DataFusion engine, and the autonomous maintenance loop.
Apache IcebergCompactionStreaming
Apache Iceberg Table Cleanup: A Production Guide
A practitioner's guide to Iceberg table cleanup — snapshot expiration, orphan file removal, manifest rewriting, delete file resolution, streaming challenges, compliance, and cost. Why sequencing matters, where teams break tables, and how to automate the full lifecycle.
CompactionApache IcebergLakeOps
Apache Iceberg rewrite_data_files: A Production Guide
The complete guide to Iceberg's rewrite_data_files procedure — strategies, parameter tuning, OOM fixes, commit conflict handling, and when to move beyond manual compaction to an automated control plane.
Apache IcebergCompactionLakeOps
Airflow & Iceberg: Building Production Maintenance DAGs
A practitioner's guide to building Airflow DAGs for Iceberg table maintenance — compaction, snapshot expiration, orphan cleanup, and manifest rewriting with working Spark SQL code. At each step, the guide contrasts the manual DAG approach with the autonomous control plane alternative, showing where the architectural boundary lies and how to migrate when the lakehouse outgrows cron-based maintenance.
Data PlatformsData LakehouseApache Iceberg
Data Lakehouse Maintenance with Airflow: Why It Breaks
Most data lakehouse teams start maintaining Iceberg tables with Airflow DAGs and Spark SQL procedures. This guide covers the five structural pitfalls that emerge at scale — fixed schedules, per-table DAGs, JVM overhead, missing coordination, and blind-spot observability — and the autonomous control plane architecture that replaces them.
Data PlatformsApache IcebergData Lakehouse
Apache Iceberg Performance Optimization: Queries to Tables
How Apache Iceberg performance actually works — the query execution pipeline, the five surfaces that degrade every production table, and the intelligent control plane that keeps file layout, sort order, metadata, and engine routing optimized continuously.
Apache IcebergData LakehouseLakeOps
Open Data Lakehouse: Build Like Google
Google engineered a multi-layered Iceberg lakehouse — autonomous storage optimization, vectorized native execution, catalog federation, and credential vending. Learn their 6-layer optimization framework and how to build the same architecture with an open, engine-neutral control plane.
Apache IcebergLakeOpsCompaction
DuckDB for Apache Iceberg
Query and write Apache Iceberg tables with DuckDB — no cluster required. Catalog setup, MERGE INTO, time travel, table layout, and when to route to DuckDB vs Spark or Trino.
Apache IcebergData LakehouseAWS
Amazon S3 Tables vs Self-Managed Iceberg
S3 Tables embeds managed Iceberg into S3 with automatic compaction. Self-managed Iceberg gives full control over catalogs, engines, and maintenance. A production comparison across compaction, observability, engine support, security, cost, and the control plane that ties it all together.
Apache IcebergLakeOpsStreaming
Apache Iceberg CDC Pipeline: Change Data Capture Best Practices
Getting CDC data into Iceberg is solved — Debezium, Flink, and DMS handle ingestion. The hard part is maintaining CDC tables that receive continuous updates and deletes. A practical guide to ingestion patterns, delete file management, and autonomous maintenance.
Apache IcebergLakeOpsCompaction
Why Your Iceberg Queries Are Slow (And How to Fix Them)
Slow Iceberg queries almost always trace back to five structural problems: small files, wrong sort order, manifest bloat, stale snapshots, or partition misalignment. This diagnostic guide shows you how to find each one, confirm it with SQL, and fix it — manually or with autonomous optimization.
Apache IcebergDelta LakeLakeOps
Apache Iceberg vs Delta Lake: A Technical Comparison
A deep technical comparison of Apache Iceberg and Delta Lake across metadata, schema evolution, partitioning, engine support, and operations — and how a control plane closes Iceberg's operational gap.
Apache IcebergLakeOpsAWS
Reduce Amazon Athena Costs on Apache Iceberg Tables
Athena charges $5 per TB scanned — and on poorly maintained Iceberg tables, every query scans far more data than it should. This guide breaks down why Athena bills explode on Iceberg (small files, bad sort order, stale manifests, scan amplification) and presents two paths to fix it: autonomous optimization with LakeOps or the manual approach with Athena SQL and Spark.
Apache IcebergLakeOpsCompaction
Replace Spark for Iceberg Compaction: Faster, Cheaper Alternatives
Spark compaction is expensive, slow, and architecturally wrong for file rewrites. Here's how to replace it with purpose-built engines that cost 90% less and finish 95% faster — plus the DIY path if you want to optimize what you already have.
Apache IcebergTrinoLakeOps
Slow Trino Queries on Iceberg? 7 Fixes for Faster Trino Iceberg Performance
Slow Trino queries on Apache Iceberg are rarely a compute problem — they're a table layout problem. Unmaintained Iceberg tables turn sub-second Trino scans into minute-long full reads. This guide covers seven proven fixes for faster Trino Iceberg performance: compaction, query-aware sorting, partition strategy, manifest optimization, lifecycle cleanup, multi-engine routing, and continuous observability. Why Trino-native maintenance falls short — and how to automate each fix at scale.
CompactionApache IcebergLakeOps
Iceberg Lake Compaction Strategies: A Practical Guide
A deep guide to bin-pack, sort, and Z-order compaction strategies for Apache Iceberg — when to use each, how to configure them, and how to automate strategy selection across hundreds of tables.
Data PlatformsApache IcebergLakehouse
Building a Lakehouse as a Service on Kubernetes with Apache Iceberg
How STACKIT built a managed lakehouse offering on Kubernetes — custom operators, CRD-driven provisioning, multi-tenant Iceberg catalogs, and the engineering lessons from bringing a sovereign lakehouse service to market in Europe.
Apache IcebergData PlatformsLakeOps
Apache Iceberg at Scale: Infrastructure, Performance, and Enterprise Lessons
Running Iceberg at 10 tables is configuration. Running it at 10,000 is infrastructure. Production lessons on infrastructure evolution, Parquet tuning, Spark configuration, catalog scaling, enterprise security, and observability-driven optimization for production Iceberg deployments.
Apache IcebergDelta LakeLakeOps
Delta Lake to Apache Iceberg: Zero-Copy Migration Without Moving Data
How to migrate Delta Lake tables to Apache Iceberg without copying or rewriting data files. Covers zero-copy metadata conversion, the mapping between Delta transaction logs and Iceberg manifest trees, Iceberg V3 spec compatibility, practical tooling (XTable, UniForm, Iceberg Delta module), and the post-migration operational discipline — compaction, sort optimization, statistics — that determines whether converted tables actually perform.
Data PlatformsApache IcebergStreaming
Iceberg CDC at Scale: How LinkedIn Ingests Billions of Upserts Per Day
LinkedIn runs Iceberg CDC on 10,000+ tables — billions of upserts daily, 2+ PB throughput. Equality vs position deletes, delete-file compaction, budgeted maintenance, and WAP branching lessons for any team running MERGE INTO at scale.
Apache IcebergLakeOpsCompaction
Iceberg Metadata at Scale: Keep Query Planning Fast on Petabyte Tables
When Iceberg metadata grows to hundreds of gigabytes, query planning — not Parquet reads — becomes the bottleneck. A practical guide for data platform teams on manifest rewriting, snapshot expiration, statistics, and metadata health at petabyte scale.
Apache IcebergStreamingApache Flink
Apache Iceberg Commit Conflicts: Causes, Prevention, and Recovery
Every concurrent write to an Apache Iceberg table risks a commit conflict. This guide covers how Iceberg's optimistic concurrency works, what triggers CommitFailedException, the common conflict scenarios in streaming and maintenance workloads, and the strategies — from partition isolation to branch-based writes — that eliminate conflicts in production.
Apache IcebergObservabilityLakeOps
Apache Iceberg Operational Runbook: Incidents, Symptoms, and Fixes
A production-ready runbook for Iceberg incidents: queries suddenly slow, planning takes minutes, write conflicts spike, storage grows uncontrolled, compaction OOMs, time travel breaks, and delete files degrade reads. Each incident follows Symptom → Root Cause → Diagnosis → Fix → Prevention.
Apache IcebergCompactionLakeOps
Automating Apache Iceberg Table Maintenance
Apache Iceberg ships the maintenance primitives — compaction, snapshot expiration, orphan cleanup, and manifest rewriting — but none of them run themselves. This guide covers why each operation matters, the correct execution order, the limitations of scripts and cron jobs, and how to automate the full lifecycle with policies, observability, and a purpose-built control plane.
CompactionApache IcebergApache Kafka
Kafka to Iceberg Compaction — Done Right
Streaming from Kafka into Apache Iceberg creates small files faster than any other write pattern. This guide covers why standard compaction approaches fail for streaming tables, how to measure compaction need, implement partition-aware compaction that avoids writer conflicts, tune rewriteDataFiles parameters, and run maintenance autonomously at scale.
Apache IcebergLakehouseCompaction
Apache Iceberg 1.11.0 — What's New?
Apache Iceberg 1.11.0 lands V3 maturity with production-ready deletion vectors, a native Variant type for semi-structured data, server-side scan planning, built-in table encryption, and a pluggable File Format API that opens the door to next-generation storage formats.
Apache IcebergAWSCompaction
AWS Glue Iceberg Optimization: A Practical Guide
AWS Glue provides native Iceberg support for cataloging, ETL, and built-in table maintenance — but production lakehouses hit limitations fast. This guide covers Glue catalog configuration, ETL best practices, compaction tuning, common pitfalls, and how a dedicated control plane fills the operational gaps.
Apache IcebergdbtCompaction
Apache Iceberg with dbt: Optimization Guide
dbt transforms your data — but who maintains the Iceberg tables underneath? A practical guide to dbt adapters, incremental strategies, table properties, and the maintenance gap that every dbt + Iceberg team hits in production.
Apache IcebergApache FlinkStreaming
Apache Iceberg with Flink: Streaming Optimization Guide
Flink streaming into Iceberg creates thousands of small files per hour. This guide covers checkpoint tuning, write distribution modes, Flink SQL patterns, and why external maintenance is essential for production streaming tables.
Apache IcebergCompactionLakeOps
Apache Iceberg Delete Files: Reducing Merge-on-Read Overhead
Delete files let Iceberg avoid rewriting data on every UPDATE or DELETE — but every unresolved delete file forces readers to reconcile at query time. A deep guide to position deletes, equality deletes, measuring overhead, and resolving accumulation before it tanks performance.
Apache IcebergPartitioningLakeOps
Apache Iceberg Table Partitioning Best Practices
Partitioning determines how much data every query must scan. Apache Iceberg's hidden partitioning and partition evolution change the game — but choosing the wrong strategy still creates performance cliffs. A practical guide to transforms, sizing, evolution, and avoiding the small-files trap.
CompactionApache IcebergLakeOps
Fixing Small Files in Apache Iceberg: A Practical Guide
Small files silently degrade every Apache Iceberg lakehouse — inflating S3 costs, slowing query planning, and bloating metadata. This guide covers root causes, measurement, manual and automated fixes, and how to eliminate the problem at scale.
Apache IcebergCompactionObservability
Apache Iceberg Table Health and Maintenance: A Complete Guide
Iceberg tables degrade silently in production — small files multiply, snapshots accumulate, orphans waste storage, and manifests fragment. A comprehensive guide to the five maintenance operations, why sequencing matters, the metrics that reveal problems early, and how to automate the full lifecycle.
Apache IcebergTrinoCompaction
Apache Iceberg with Trino: Performance Optimization Guide
A practical guide to optimizing Apache Iceberg queries and table maintenance with Trino — covering scan planning, predicate pushdown, file pruning, Trino-side tuning, maintenance procedures, physical layout optimization, and how a dedicated control plane eliminates JVM overhead while adding cross-engine intelligence.
FinOpsApache IcebergLakeOps
State of Iceberg FinOps and Cost Reduction in 2026
State of Iceberg FinOps in 2026: where lakehouse spend leaks, what to measure, how autonomous management and optimization are replacing manual maintenance — and a practical survey of tools from cloud optimizers to control planes.
Apache IcebergData PlatformsData Lake
Iceberg Lake for Data Analytics: Optimization Guide
Eight optimization layers for data platform engineers running BI, ad-hoc SQL, and aggregation pipelines on Apache Iceberg — from partition design and file sizing through compaction, routing, and continuous maintenance.
Apache IcebergLakeOpsCloud Cost
7 Iceberg Lakehouse Cost Reduction Strategies
Iceberg lakehouses silently accumulate cost from small files, dead snapshots, orphan data, unoptimized layouts, and over-provisioned compute. Seven practical strategies — from deploying an autonomous control plane to leveraging partition evolution — that production data teams use to cut lakehouse spend by up to 80%.
Apache IcebergLakeOpsAnalytics
Optimizing Iceberg Lakehouse Performance
Iceberg tables degrade silently — small files from streaming, unsorted data, fragmented manifests, accumulated delete files. Each one caps query speed regardless of engine. Six concrete optimization layers, how they interact, and how autonomous maintenance keeps every table at peak performance.
CompactionData LakehouseApache Iceberg
9 Iceberg Table Compaction Tools Compared for Production Lakehouses
Compaction keeps Apache Iceberg lakehouses fast and lean — but every tool approaches it differently. A side-by-side look at nine production options: LakeOps, AWS Glue, Amazon S3 Tables, Snowflake, Google BigLake, Cloudera, Starburst, Dremio, and Databricks.
Apache IcebergLakeOpsLakehouse
Iceberg Lakehouse Optimization with LakeOps
A practical walkthrough of optimizing an Apache Iceberg lakehouse end to end — from connecting catalogs and diagnosing table health through autonomous compaction, lifecycle management, and multi-engine routing to measurable cost and performance outcomes.
CompactionApache IcebergLakehouse
Optimizing Iceberg Lake Compaction: A Guide
Compaction is the most impactful operation in an Apache Iceberg lakehouse — and the hardest to get right at scale. File merging is the easy part. Knowing when to trigger it, what sort strategy to apply per table, how to avoid conflicting with other maintenance, and how to do it without spinning up expensive JVM clusters — that is the real problem. A breakdown of what modern compaction actually requires.
Apache IcebergLakehouseLakeOps
Iceberg Lakehouse Optimization — The Right Way
Apache Iceberg gives your lakehouse warehouse-grade reliability on object storage — but the format does not optimize itself. A practical guide to every operational pillar a production Iceberg lakehouse needs — from lake-wide observability and query-aware compaction to snapshot lifecycle, metadata health, and governance — and how LakeOps runs it all from a single control plane.
Apache IcebergLakeOpsFinOps
Autonomous Iceberg Table Maintenance for Data Lakes
Iceberg tables need continuous maintenance — compaction, snapshot expiration, manifest optimization, and orphan cleanup — but manual scripts break at scale. A deep look at what autonomous table maintenance means in practice: how telemetry-driven orchestration replaces reactive firefighting and keeps every table healthy without human intervention.
Apache IcebergLakeOpsCloud Cost
Apache Iceberg Cost Optimization in 2026
Your Iceberg lake is overcharging you from four directions at once — storage bloat, query compute waste, compaction overhead, and engineering time. This post breaks down exactly where each dollar goes and how autonomous table management eliminates the waste without touching your pipelines.
Apache IcebergCompactionLakeOps
Benchmarking Lakeops: A Production-Grade Compaction Engine for Apache Iceberg
How we compacted 4.5 TB across 10 real production tables, achieved up to 99.8% file reduction, and made Apache Spark OOM on a job we finished in 11 minutes.
Apache IcebergCompactionLakeOps
Building a Distributed Compaction Engine for Apache Iceberg with Rust + DataFusion
How we built a high-performance, distributed compaction engine for Apache Iceberg using Rust and DataFusion—architecture, design choices, and lessons learned.
Apache IcebergCompactionData Platforms
Cracking the Ice: The Battle Between Sort and Binpack in Apache Iceberg
Unlocking performance vs. optimizing storage — choosing the right compaction strategy for your data lake.