# Alex Merced's Lakehouse Blog > Alex Merced's Lakehouse Blog (iceberglakehouse.com) is Alex Merced's independent blog on the open lakehouse. It has 480 articles on Apache Iceberg internals, lakehouse catalogs such as Apache Polaris and the Iceberg REST Catalog, table maintenance and performance, query engines, and the agentic lakehouse, where AI agents query governed data through semantic layers and open protocols such as MCP. Alex is Head of Developer Relations at Dremio and a co-author of Apache Iceberg: The Definitive Guide. The pillar pages below are the best starting points; the posts are listed newest first within each topic. ## Pillar Pages - [Agentic Analytics and the Semantic Layer: Natural Language to SQL on Iceberg](https://iceberglakehouse.com/agentic-analytics/): Agentic analytics lets AI agents answer business questions by generating and executing SQL against governed lakehouse data. This guide explains how NL2SQL works, what role the semantic layer plays in accuracy, and how to build a production natural language analytics system on Apache Iceberg. - [What Is an Agentic Lakehouse | Architecture, Governance, and AI Agents](https://iceberglakehouse.com/agentic-lakehouse/): An agentic lakehouse is an architecture where AI agents can safely query, reason over, and act on governed lakehouse data. Learn what the four required layers are, how governance and trust work, and why open standards matter for AI-ready data infrastructure. - [Apache Iceberg Architecture: Metadata Tree, Snapshots, and Catalogs Explained](https://iceberglakehouse.com/apache-iceberg-architecture/): A technical deep dive into how Apache Iceberg works internally — the metadata JSON, manifest list, manifest files, data files, commit flow, query planning, and concurrency model. - [Iceberg REST Catalog and Apache Polaris: Multi-Engine Catalog Guide](https://iceberglakehouse.com/apache-iceberg-rest-catalog/): The Iceberg REST Catalog specification is an open HTTP API that lets any engine connect to any catalog. This guide explains how the spec works, what Apache Polaris provides, how credential vending works, and how to connect Spark, Trino, and PyIceberg to a REST catalog. - [Apache Iceberg Schema Evolution and Hidden Partitioning Explained](https://iceberglakehouse.com/apache-iceberg-schema-evolution/): Explore the inner workings of Apache Iceberg schema evolution, unique column ID tracking, hidden partitioning transforms, partition evolution, and Dremio performance optimizations. - [Apache Iceberg Snapshots and Time Travel: How Table History Works](https://iceberglakehouse.com/apache-iceberg-snapshots-and-time-travel/): Every Apache Iceberg commit creates an immutable snapshot. This guide explains the snapshot model, how time travel queries work, how to use table branches and tags, and how to manage snapshot retention. - [Apache Iceberg vs Delta Lake vs Apache Hudi: Complete Comparison](https://iceberglakehouse.com/apache-iceberg-vs-delta-lake-vs-hudi/): A neutral, technical comparison of Apache Iceberg, Delta Lake, and Apache Hudi covering architecture, multi-engine support, streaming, governance, and a decision framework by workload type. - [Apache Iceberg Explained: Architecture, Snapshots, and Catalogs](https://iceberglakehouse.com/apache-iceberg/): Apache Iceberg is an open table format for huge analytic tables. Learn how it works internally, from the metadata tree and snapshot model to hidden partitioning, schema evolution, and the catalog API that makes multi-engine access possible. - [Open Table Formats Benchmark: Iceberg vs Delta Lake vs Hudi](https://iceberglakehouse.com/benchmarks/open-table-formats/): An exhaustive, production-grade performance benchmark of open table formats. Analyze query scan speed, write throughput, maintenance overhead, and engine selection caveats across Apache Iceberg, Delta Lake, and Apache Hudi. - [Books by Alex Merced](https://iceberglakehouse.com/books/): Books on Apache Iceberg, lakehouse architecture, catalogs, and agentic analytics, written by Alex Merced. - [Data Lakehouse vs Data Lake vs Data Warehouse: Key Differences](https://iceberglakehouse.com/data-lakehouse-vs-data-lake-vs-data-warehouse/): A practical comparison of the three main data storage architectures — data lake, data warehouse, and data lakehouse — covering storage model, query performance, governance, cost, AI/ML support, and when each is the right choice. - [What Is a Data Lakehouse | Apache Iceberg & Lakehouse Guide](https://iceberglakehouse.com/data-lakehouse/): A data lakehouse combines the low-cost storage of a data lake with the reliability and query performance of a data warehouse. Learn how the architecture works, what layers it includes, and when it makes sense to build one. - [Lakehouse for AI Agents: Architecture and Implementation Guide](https://iceberglakehouse.com/lakehouse-for-ai-agents/): AI agents need governed, documented, queryable data to work reliably. This guide explains how to build a data architecture that serves AI agents through structured interfaces, covering the data layer, catalog, semantic layer, and agent connection patterns. - [Open Table Formats Explained: Iceberg, Delta Lake, Hudi, and Paimon](https://iceberglakehouse.com/open-table-formats/): Open table formats add ACID transactions, schema evolution, and query optimization to raw object storage. This guide explains what they are, how they work, and how Apache Iceberg, Delta Lake, Apache Hudi, and Apache Paimon each approach the problem. ## Articles: Apache Iceberg and Table Formats - [Keeping Audit Snapshots Alive While Iceberg Snapshot Expiration Runs Every Night](https://iceberglakehouse.com/posts/iceberg-tags-regulatory-snapshot-retention/) (2026-09-28): How Iceberg snapshot tags keep audit snapshots alive through nightly expiration: retention calendars, RETAIN semantics, compaction cost, and erasure conflicts. - [Why the Iceberg DataFusion Integration Is Moving to Apache DataFusion](https://iceberglakehouse.com/posts/datafusion-iceberg-moves-to-apache-datafusion/) (2026-09-21): Why the Iceberg DataFusion integration moved to the DataFusion project, and what the split means for users, Comet, and iceberg-rust contributors. - [What Iceberg v4's Proposed FILE Type Means for Multimodal Tables](https://iceberglakehouse.com/posts/iceberg-v4-file-type-multimodal-tables/) (2026-09-21): Iceberg v4's proposed FILE type brings first-class media references to tables, via Parquet's FILE logical type, ranges, checksums, and pre-signed URLs. - [CVE-2026-73334 and the Trust Boundary Inside an Encrypted Parquet File](https://iceberglakehouse.com/posts/parquet-kms-url-cve-2026-73334-iceberg/) (2026-09-21): CVE-2026-73334 lets a tampered Parquet footer route a reader's KMS token to an attacker. Here's the fix, Iceberg's safe path, and how to audit your lakehouse. - [Parquet Page Indexes and the Last Mile of Pruning in Apache Iceberg](https://iceberglakehouse.com/posts/parquet-page-index-pruning-iceberg/) (2026-09-21): Parquet page indexes can cut selective Iceberg scans by an order of magnitude on sorted data. How they work, what they cost, and how to lay out tables. - [How Apache Polaris Plans to Share Iceberg Tables Across Organizations](https://iceberglakehouse.com/posts/polaris-open-sharing-iceberg-tables/) (2026-09-21): Apache Polaris's Open Sharing proposal adds first-class shares, external consumers, and listings so any Iceberg REST engine can read shared tables. - [Keeping Lakehouse Traffic Off the Public Internet With Apache Polaris](https://iceberglakehouse.com/posts/polaris-private-networking-lakehouse/) (2026-09-21): A private Polaris lakehouse still leaks data if the storage hop goes public. How to bind vended credentials to private networks on AWS, Azure, and GCP. - [Vector Search Directly Over Iceberg Tables, and When You Still Need a Vector Database](https://iceberglakehouse.com/posts/vector-search-over-iceberg-tables/) (2026-09-21): Embeddings are just columns. When exact vector search over Iceberg scans beats a vector database, when it does not, and how to lay out tables. - [Running Apache Polaris in Production](https://iceberglakehouse.com/posts/apache-polaris-in-production/) (2026-09-10): Apache Polaris past the quickstart: persistence backends, realm bootstrap, replica token signing, upgrades, backups, and which failures take the lakehouse offline. - [Migrating Into Iceberg Without Moving Data](https://iceberglakehouse.com/posts/iceberg-in-place-migration/) (2026-09-10): add_files, snapshot, and migrate compared: the three in-place paths into Iceberg, the reconciliation each requires, the layout traps, and the rollback story. - [What Iceberg Table Maintenance Actually Costs](https://iceberglakehouse.com/posts/iceberg-maintenance-budget/) (2026-09-10): A cost model for compaction, snapshot expiry, orphan cleanup, and manifest rewriting: what each operation spends, on which meter, and how to set a schedule. - [Partition Statistics Files in Apache Iceberg](https://iceberglakehouse.com/posts/iceberg-partition-statistics-files/) (2026-09-10): The underused Iceberg metadata for planning: what the partition statistics file holds, what the spec guarantees, how to write one, and when it earns its slot. - [The 2026 Iceberg REST Catalog Compatibility Report](https://iceberglakehouse.com/posts/iceberg-rest-catalog-compatibility-report-2026/) (2026-09-10): A repeatable test for what an Iceberg REST catalog actually serves, a scoring scheme that separates design from breakage, and the 2026 evidence across seven catalogs. - [Serving Iceberg Tables From Two Regions](https://iceberglakehouse.com/posts/iceberg-two-region-serving/) (2026-09-10): Three multi-region topologies that work and one that mostly does not, what an Iceberg commit costs across regions, and where the catalog has to live. - [Disaster Recovery for Iceberg Tables: Replication, Backup, and Restore](https://iceberglakehouse.com/posts/disaster-recovery-for-iceberg-tables/) (2026-09-02): Disaster recovery for Iceberg across four tiers: snapshots, object versioning, catalog backup, and cross-region replication. - [Geospatial Data in Apache Iceberg: Geometry, Geography, and GeoParquet](https://iceberglakehouse.com/posts/geospatial-data-in-apache-iceberg/) (2026-09-02): How Iceberg v3 geometry and geography types, bounding boxes, and native Parquet types give spatial data first-class standing. - [Default Column Values and Field IDs: How Iceberg Schema Evolution Works at the Spec Level](https://iceberglakehouse.com/posts/iceberg-default-values-and-field-ids/) (2026-09-02): How field IDs and initial and write defaults let Iceberg change schemas on large tables without rewriting data, at the spec level. - [The Iceberg Table Properties That Actually Matter](https://iceberglakehouse.com/posts/iceberg-table-properties-that-matter/) (2026-09-02): The Iceberg table properties that decide file count, pruning, write amplification, retention, and metadata growth, by workload. - [Inside the Puffin File Format](https://iceberglakehouse.com/posts/inside-the-puffin-file-format/) (2026-09-02): The Puffin file format inside out, byte by byte, covering Theta sketches for distinct values and deletion vectors. - [Local Iceberg Development Environments: Docker, MinIO, and In-Memory Catalogs for CI](https://iceberglakehouse.com/posts/local-iceberg-development-environments/) (2026-09-02): Local Iceberg development environments: in-process catalogs, a Docker Compose stack with MinIO, and CI configurations that run either. - [Moving Iceberg Tables Between Catalogs Without Rewriting Data](https://iceberglakehouse.com/posts/moving-iceberg-tables-between-catalogs/) (2026-09-02): Why moving Iceberg tables between catalogs is a pointer copy, and the protocol that makes a cutover safe for one table or thousands. - [Storage-Partitioned Joins and the Bucket Transform](https://iceberglakehouse.com/posts/storage-partitioned-joins-and-the-bucket-transform/) (2026-09-02): How the spec-defined bucket transform lets engines skip the shuffle in joins, and how to set it up and keep it engaged in Spark. - [Agent-Driven Storage Tiering for Apache Iceberg: Moving Cold Data Without Breaking Queries](https://iceberglakehouse.com/posts/agent-driven-iceberg-storage-tiering/) (2026-08-25): A background agent can move cold Iceberg partitions to cheaper tiers without breaking live queries. Heatmaps, path-safe moves, and restore paths. - [DataFusion Comet 1.0 and What Native Rust Scans Change for Spark on Iceberg](https://iceberglakehouse.com/posts/datafusion-comet-1-spark-iceberg/) (2026-08-25): DataFusion Comet 1.0 replaces Spark Iceberg scans with native Rust. What speeds up, what still falls back to the JVM, and how to deploy it. - [FSST and ALP: The Two Encodings Fixing Parquet's Weakest Compression Cases](https://iceberglakehouse.com/posts/fsst-alp-parquet-encodings/) (2026-08-25): ALP and FSST target Parquet's worst cases: floats and high-cardinality strings. How they work and what they change for Iceberg tables. - [High-Throughput Branch Merging: Automating Concurrency and Conflict Resolution in Multi-Branch Iceberg Pipelines](https://iceberglakehouse.com/posts/multi-branch-iceberg-merge-automation/) (2026-08-25): High-throughput Iceberg branch merges need conflict detection and automation. How to reconcile concurrent writes without stalling pipelines. - [Multi-Cloud REST Catalog Topologies: Running Apache Polaris Across AWS, Azure, and GCP](https://iceberglakehouse.com/posts/multi-cloud-polaris-rest-catalog-topologies/) (2026-08-25): Polaris can catalog Iceberg tables across AWS, Azure, and GCP. Four topologies, credential vending, and the tradeoffs of each design. - [Parquet-Only Manifests in Iceberg v4: Why the Metadata Layer Is Going Columnar](https://iceberglakehouse.com/posts/parquet-manifests-iceberg-v4/) (2026-08-25): Iceberg v4 is moving manifests from Avro to Parquet so planners can read only the stats they need. Why the metadata layer is going columnar. - [Serverless Iceberg Ingestion with PyIceberg and DuckDB: Micro-Batches Without a Spark Cluster](https://iceberglakehouse.com/posts/serverless-iceberg-microbatch-pyiceberg-duckdb/) (2026-08-25): Land small Iceberg micro-batches with PyIceberg and DuckDB in a serverless function. Commits, concurrency, and why Spark is the wrong default. - [Zero-Copy Warehouse Modernization: Migrating Legacy Databases to Apache Iceberg Without Downtime](https://iceberglakehouse.com/posts/zero-copy-legacy-db-to-iceberg/) (2026-08-25): Move a legacy warehouse to Iceberg without downtime by virtualizing first. Consumer cutover, parity checks, and background copy without double-ETL. - [The Hidden Cost of Tiny Iceberg Commits](https://iceberglakehouse.com/posts/hidden-cost-of-tiny-iceberg-commits/) (2026-08-24): Trace what one tiny Iceberg commit writes, then model hourly, per-minute, and per-second cadences so streaming costs become arithmetic, not adjectives. - [Deletion Vectors vs Position Deletes vs Equality Deletes: The Iceberg Delete Story in 2026](https://iceberglakehouse.com/posts/iceberg-delete-story-2026/) (2026-08-24): Position deletes, equality deletes, and deletion vectors compared from the Iceberg spec: what each writes, how readers apply it, and when to use which. - [Iceberg Is Becoming a Library, Not Just a Table Format](https://iceberglakehouse.com/posts/iceberg-is-becoming-a-library/) (2026-08-24): Iceberg is turning from a JVM table format into a library other systems embed. What that shift changes for engines, catalogs, and the spec itself. - [Iceberg Is Escaping the JVM: Why Rust, Go, Python and C++ Implementations Matter](https://iceberglakehouse.com/posts/iceberg-is-escaping-the-jvm/) (2026-08-24): Rust, Go, Python, and C++ Iceberg implementations change who can write the format. Why multi-language clients matter more than another JVM engine. - [The Iceberg REST Catalog Compatibility Test: One Suite of Operations Every Platform Should Pass](https://iceberglakehouse.com/posts/iceberg-rest-catalog-compatibility-test/) (2026-08-24): One suite of REST catalog operations every Iceberg platform should pass. What sameness means, where implementations diverge, and how to test it. - [Iceberg REST Remote Scan Planning Changes More Than Query Performance](https://iceberglakehouse.com/posts/iceberg-rest-remote-scan-planning/) (2026-08-24): Remote scan planning moves Iceberg file selection into the catalog. What that changes for engines, governance, and operational cost beyond query speed. - [Iceberg Row Lineage: The Feature AI and CDC Workloads Will Eventually Depend On](https://iceberglakehouse.com/posts/iceberg-row-lineage/) (2026-08-24): Iceberg row lineage gives rows a durable identity across rewrites. Why CDC pipelines and AI workloads will eventually depend on it. - [Iceberg v4's Adaptive Metadata Tree, Explained From First Principles](https://iceberglakehouse.com/posts/iceberg-v4-adaptive-metadata-tree/) (2026-08-24): Iceberg v4's adaptive metadata tree, explained from first principles: why commits rewrite too much today and how the tree makes change cheaper. - [Why Iceberg v4 Is Really About Making the Cost of Change Proportional to the Change](https://iceberglakehouse.com/posts/iceberg-v4-cost-of-change-principle/) (2026-08-24): Iceberg v4 is really about making the cost of a change proportional to the change. The principle, the current tax, and what the redesign pays down. - [The Catalog Can Now Plan Your Iceberg Query: Inside REST Scan Planning](https://iceberglakehouse.com/posts/inside-iceberg-rest-scan-planning/) (2026-08-24): A mechanics walkthrough of Iceberg REST scan planning: client-side planning, remote endpoints, pagination, and where engine support stands in 2026. - [Can Seven Different Iceberg REST Catalogs Really Run the Same DuckDB Code?](https://iceberglakehouse.com/posts/seven-rest-catalogs-one-duckdb-script/) (2026-08-24): Can seven Iceberg REST catalogs run the same DuckDB script? What the protocol makes portable, what still differs, and a test matrix you can rerun. - [Stop Flattening Your JSON: How Iceberg Variant Changes Semi-Structured Analytics](https://iceberglakehouse.com/posts/stop-flattening-your-json-iceberg-variant/) (2026-08-24): Iceberg Variant stores JSON as navigable binary with shredding for columnar filters. Why flattening wide tables is no longer the only performance path. - [What Actually Happens When Two Engines Write the Same Iceberg Table at Once?](https://iceberglakehouse.com/posts/two-engines-one-iceberg-table/) (2026-08-24): What happens when two engines write the same Iceberg table at once: snapshot isolation, optimistic commits, conflict detection, and when retries fail. - [Variant Shredding Explained: How Iceberg Gets Columnar Performance From Messy JSON](https://iceberglakehouse.com/posts/variant-shredding-explained/) (2026-08-24): Variant shredding turns messy JSON into Parquet columns with statistics. How the layout works, how readers reassemble values, and why some queries prune. - [Who Actually Owns an Iceberg Table? Managed, External, and the New Vocabulary of Lakehouse Control](https://iceberglakehouse.com/posts/who-owns-an-iceberg-table/) (2026-08-24): Managed and external Iceberg tables mean different things on every platform. Five ownership dimensions and a translation method for vendor vocabulary. - [Mastering Apache Iceberg v3 Deletion Vectors for High-Throughput Streaming Ingest](https://iceberglakehouse.com/posts/apache-iceberg-v3-deletion-vectors-streaming-ingest/) (2026-08-19): Apache Iceberg v3 deletion vectors for high-throughput streaming ingest: how bitmaps and Puffin files fix CDC write amplification and read decay. - [The Decoupled Data Lakehouse: Multi-Engine Freedom with Open REST Catalogs](https://iceberglakehouse.com/posts/decoupled-data-lakehouse-multi-engine-rest-catalogs/) (2026-08-19): The decoupled data lakehouse: multi-engine freedom with open REST catalogs, credential vending, and an estate that outlives its tools. - [Building Lightweight Serverless Ingestion to Apache Iceberg with PyIceberg and DuckDB](https://iceberglakehouse.com/posts/serverless-iceberg-ingestion-pyiceberg-duckdb/) (2026-08-19): Build lightweight serverless ingestion to Apache Iceberg with PyIceberg and DuckDB, running small feeds in functions that bill for seconds. - [Write-Audit-Publish with Apache Iceberg Branches: CI/CD for Your Data](https://iceberglakehouse.com/posts/write-audit-publish-iceberg-branches/) (2026-08-19): Write-Audit-Publish with Apache Iceberg branches brings CI/CD to your data: stage, audit, then fast-forward to main so consumers never see unvalidated bytes. - [Zero-Copy Warehouse Modernization: Moving to Apache Iceberg Without Downtime](https://iceberglakehouse.com/posts/zero-copy-warehouse-modernization-iceberg/) (2026-08-19): A practical guide to modernizing a data warehouse to Apache Iceberg without downtime, using federation first, then redirecting new data, then. - [Autonomous Table Optimization When Your Query Workload Stops Being Predictable](https://iceberglakehouse.com/posts/autonomous-table-optimization/) (2026-08-04): Autonomous table optimization when query workloads stop being predictable: observing file layout and query patterns, scoring compaction work, adaptive. - [Surviving Optimistic Commit Collisions When Hundreds of Agents Write to Iceberg](https://iceberglakehouse.com/posts/high-concurrency-agent-writes-iceberg/) (2026-08-04): Surviving optimistic commit collisions when hundreds of agents write to Iceberg: which conflicts are real, commit buffers, partitioning, and the patterns. - [How Iceberg V3 Deletion Vectors Fixed Merge-on-Read for Streaming Tables](https://iceberglakehouse.com/posts/iceberg-v3-deletion-vectors-merge-on-read/) (2026-08-04): How Iceberg V3 deletion vectors replaced accumulating positional delete files and made merge-on-read viable for streaming and CDC tables. - [How Iceberg V3 Variant Shredding Changed Semi-Structured Data on S3 Tables](https://iceberglakehouse.com/posts/iceberg-v3-variant-shredding-s3-tables/) (2026-08-04): How Iceberg V3's Variant type and Parquet shredding turn JSON columns into prunable typed columns, with real benchmark tradeoffs and a migration path. - [Reading the Apache Iceberg V4 Proposals Before They Land](https://iceberglakehouse.com/posts/iceberg-v4-roadmap/) (2026-08-04): A field guide to the Apache Iceberg V4 proposals: adaptive metadata trees, single-file commits, typed statistics, column families, and what is safe. - [Cross-Cloud Credential Vending in Apache Polaris and the End of Permanent Storage Keys](https://iceberglakehouse.com/posts/polaris-cross-cloud-credential-vending/) (2026-08-04): How Apache Polaris vends short-lived, prefix-scoped storage credentials across AWS, Azure, and GCP, and how to retire permanent storage keys for good. - [How the Iceberg REST Catalog Turned Into the Lakehouse Control Plane](https://iceberglakehouse.com/posts/rest-catalog-v2-multi-table-commits/) (2026-08-04): How the Iceberg REST catalog became the lakehouse control plane: multi-table atomic commits, credential vending, capability negotiation, and what still breaks. - [What Zero-Copy Data Sharing Actually Does Between Salesforce, Snowflake, and Databricks](https://iceberglakehouse.com/posts/zero-copy-data-sharing-iceberg/) (2026-08-04): What zero-copy data sharing actually does across Salesforce, Snowflake, and Databricks: query federation, file federation, catalog federation, and when. - [Apache Polaris 1.7.0 and the Quiet Work of Making a Catalog Trustworthy](https://iceberglakehouse.com/posts/apache-polaris-1-7-0/) (2026-08-02): Apache Polaris 1.7.0 deep dive: idempotent writes, semantic models, stricter credential vending, orphan cleanup, and what the upgrade asks of you. - [Designing Batch Pipelines That Write Well Into Apache Iceberg](https://iceberglakehouse.com/posts/batch-pipelines-into-apache-iceberg/) (2026-08-02): How to design batch pipelines that write well into Apache Iceberg: commit strategy, partitioning, sort order, write-audit-publish, and maintenance done right. - [Apache Iceberg Support Across the Major Hyperscalers](https://iceberglakehouse.com/posts/iceberg-across-the-hyperscalers/) (2026-08-02): How AWS, Google Cloud, and Microsoft Azure actually support Apache Iceberg: storage, catalogs, maintenance, governance, and interoperability, layer by layer. - [Running an Apache Iceberg Lakehouse With No Internet Connection](https://iceberglakehouse.com/posts/air-gapped-iceberg-lakehouse/) (2026-07-28): A practical guide to deploying an Iceberg lakehouse in air-gapped environments: component choices, artifact pipelines, identity without a cloud. - [When the Query Optimizer Starts Managing Its Own Materializations](https://iceberglakehouse.com/posts/autonomous-reflections-agentic-lakehouse/) (2026-07-28): Autonomous materialized view management replaces quarterly review meetings with workload-driven scoring, and it's essential when AI agents generate. - [Why Iceberg V4 Wants to Retire Equality Deletes, and What Streaming Teams Should Do About It](https://iceberglakehouse.com/posts/equality-deletes-iceberg-v4/) (2026-07-28): Equality deletes made streaming upserts into Iceberg practical at the cost of read performance. - [Apache Fluss and Kafka Solve Different Problems in an Iceberg Pipeline](https://iceberglakehouse.com/posts/fluss-kafka-iceberg-streaming/) (2026-07-28): Fluss puts a columnar, indexed hot tier between Kafka and Iceberg. Here's what it changes structurally, what Kafka still does better, and how to benchmark. - [Serving Sub-Second Queries Over an Iceberg Lakehouse With a Hot Tier](https://iceberglakehouse.com/posts/hot-tier-iceberg-serving/) (2026-07-28): A lakehouse cannot serve sub-second queries over seconds-old data. A hot tier in front solves it, with consequences for consistency, governance. - [Surviving Commit Conflicts When Dozens of Writers Hit the Same Iceberg Table](https://iceberglakehouse.com/posts/iceberg-concurrent-commits-agents/) (2026-07-28): Commit conflicts multiply with writer count, and AI agents introduce unpredictable write patterns. - [The Jackson 3 Problem in Apache Iceberg, and What It Means for Your Code](https://iceberglakehouse.com/posts/iceberg-jackson-3-migration/) (2026-07-28): Jackson 3 changes everything: package names, unchecked exceptions, flipped defaults. Here's what breaks, why the engines are fine and your service isn't. - [Wiring an AI Agent to Apache Polaris with the Model Context Protocol](https://iceberglakehouse.com/posts/mcp-apache-polaris/) (2026-07-28): The catalog is the right attachment point for AI agents working against a lakehouse. Here's how to wire the official Polaris MCP Server and add the read. - [Governing Iceberg Tables Across Regions Without Three Sets of Permissions](https://iceberglakehouse.com/posts/multi-region-catalog-federation/) (2026-07-28): Catalog federation gives you one authorization model and one audit point across regions. Here's what it solves, what it doesn't, and how to build. - [Federating Oracle With an Open Lakehouse Instead of Migrating It](https://iceberglakehouse.com/posts/oracle-lakehouse-federation/) (2026-07-28): Federate first so analytics work now, migrate what benefits from migrating, and leave the rest where it is indefinitely. - [The Parquet Versioning Problem, and Why Iceberg Cares About It](https://iceberglakehouse.com/posts/parquet-versioning-iceberg-v4/) (2026-07-28): Parquet files have a version field that doesn't reliably signal feature requirements. A new versioning discipline is coming, borrowing from Iceberg's. - [Building Iceberg Pipelines in Python Without Standing Up Spark](https://iceberglakehouse.com/posts/python-native-iceberg-pipelines/) (2026-07-28): A large share of production transformations fit comfortably on one machine. PyIceberg, DuckDB, and branch isolation give you a production path that debugs. - [Apache Polaris and Multi-Engine Iceberg Catalogs](https://iceberglakehouse.com/posts/apache-polaris-asf-top-level-multi-engine-catalog-interoperability/) (2026-07-13): An in-depth exploration of apache polaris and multi-engine iceberg catalogs - [Iceberg v3 Deletion Vectors and Faster DML](https://iceberglakehouse.com/posts/iceberg-v3-deletion-vectors-merge-on-read-dml-performance/) (2026-07-13): An in-depth exploration of iceberg v3 deletion vectors and faster dml - [Iceberg Variant Type for AI JSON Data](https://iceberglakehouse.com/posts/iceberg-variant-type-semi-structured-ai-datasets/) (2026-07-13): An in-depth exploration of iceberg variant type for ai json data - [Iceberg Concurrency for AI Agent Writes](https://iceberglakehouse.com/posts/llm-agency-concurrency-isolation-high-frequency-agent-writes-iceberg/) (2026-07-13): An in-depth exploration of iceberg concurrency for ai agent writes - [REST Catalog Credential Vending for Iceberg](https://iceberglakehouse.com/posts/rest-catalog-credential-vending-zero-copy-security/) (2026-07-13): An in-depth exploration of rest catalog credential vending for iceberg - [REST Catalog V2: Fixing Iceberg Protocol Debt](https://iceberglakehouse.com/posts/rest-catalog-v2-standard-iceberg-scaling-protocol-debt/) (2026-07-13): An in-depth exploration of rest catalog v2: fixing iceberg protocol debt - [The State of Apache Iceberg v4 in July 2026: What the Dev List Tells Us About the Format's Next Chapter](https://iceberglakehouse.com/posts/iceberg-v4-state-july-2026/) (2026-07-06): What the Iceberg v4 dev list tells us about adaptive metadata trees, single-file commits, column updates, and the format's next chapter in mid-2026. - [The State of Apache Arrow in 2026: Ten Years In, the Invisible Standard Is Everywhere](https://iceberglakehouse.com/posts/state-of-apache-arrow-2026/) (2026-07-06): Apache Arrow at 10, ADBC, Flight SQL, nanoarrow, the AI reinterpretation, and how an in-memory standard eliminated the copy tax across the data stack. - [The State of Apache Parquet in 2026: The Quiet Format Enters Its Loudest Decade](https://iceberglakehouse.com/posts/state-of-apache-parquet-2026/) (2026-07-06): Apache Parquet in 2026, variant types, geospatial, ALP encoding, footer redesign, the versioning debate, and how the decade-old format is renovating. - [The State of Apache Polaris in July 2026: From Incubating Catalog to the Governance Layer of the Open Lakehouse](https://iceberglakehouse.com/posts/state-of-apache-polaris-july-2026/) (2026-07-06): Apache Polaris as a TLP, federation, credential vending, semantic layers, lineage, and how the open catalog became the governance plane. - [The State of Streaming to Apache Iceberg in July 2026: Every Path, Its Latency, and What to Do When Seconds Are Not Fast Enough](https://iceberglakehouse.com/posts/streaming-to-iceberg-july-2026/) (2026-07-06): Every path for streaming data into Iceberg in 2026, Flink, Spark, Kafka Connect, broker-native, managed pipelines, with honest latency numbers. - [High-Performance Columnar Transfers: Combining Apache Arrow Flight and Iceberg REST Catalogs](https://iceberglakehouse.com/posts/arrow-flight-iceberg-rest-catalog-columnar-transfers/) (2026-07-06): Modern lakehouse architecture is easier to reason about when you separate two questions. The first question is how a system discovers and governs a... - [Conversational AI on Managed Iceberg: Exposing Amazon S3 Tables through MCP](https://iceberglakehouse.com/posts/conversational-ai-managed-iceberg-s3-tables-mcp-server/) (2026-07-06): The most interesting part of conversational analytics is not the chat box. The chat box is just the surface area. The harder question is what happe... - [Decoupled Catalogs vs. Managed Tables: Architectural Freedom in the Age of Table Format Convergence](https://iceberglakehouse.com/posts/decoupled-catalogs-vs-managed-tables-table-format-convergence/) (2026-07-06): Open table formats have changed buyer expectations. A few years ago, the question was whether an organization should put more analytical data into... - [Implementing Positional Deletes in Iceberg v3: Streamlining Merge-on-Read for Fast-Inbound Event Lakes](https://iceberglakehouse.com/posts/iceberg-v3-positional-deletes-deletion-vectors-event-lakes/) (2026-07-06): Event data has a way of humbling neat architecture diagrams. It arrives late. It arrives twice. It arrives with incorrect attributes. It needs priv... - [Mapping the Variant Type in Iceberg v3: Standardizing Semi-Structured AI JSON Payloads](https://iceberglakehouse.com/posts/iceberg-v3-variant-type-ai-json-payloads/) (2026-07-06): AI applications are messy data producers. They create prompts, completions, tool calls, retrieval traces, ranking signals, evaluation scores, safet... - [Multi-Engine Catalog Federation with Apache Polaris: Syncing Google Cloud, AWS, and Azure Metadata](https://iceberglakehouse.com/posts/multi-engine-catalog-federation-apache-polaris-multicloud/) (2026-07-06): Open table formats changed the data lakehouse conversation, but they did not finish it. A table can be stored in an open format and still be hard t... - [Designing Private, Air-Gapped Data Lakehouses: Scaling Iceberg in Highly Secure, On-Premises Clouds](https://iceberglakehouse.com/posts/private-air-gapped-data-lakehouses-iceberg-secure-clouds/) (2026-07-06): Some of the most important lakehouse work happens in environments that will never look like a simple public-cloud reference architecture. Defense... - [Lakehouse Table Formats in 2026: Iceberg, Delta Lake, Hudi, Paimon, and DuckLake, How They Work, Where They Stand, and Where They're Going](https://iceberglakehouse.com/posts/table-formats-2026-breakdown/) (2026-07-06): The table format war is over, and the table formats are not. Both halves of that sentence are true, both matter, and the tension between them is ex... - [Mastering Apache Iceberg v3: What's New and How to Plan Your Upgrade](https://iceberglakehouse.com/posts/apache-iceberg-v3-upgrade/) (2026-05-28): Apache Iceberg v3 adds deletion vectors, VARIANT type, row lineage, and table encryption. Here's what changed and how to plan your upgrade. - [How Apache Iceberg Resolves the Hybrid-Cloud Challenge in Heavily Regulated Markets](https://iceberglakehouse.com/posts/iceberg-hybrid-cloud-regulated-markets/) (2026-05-28): Apache Iceberg gives regulated enterprises data sovereignty with hybrid-cloud deployments. Learn how open catalogs and Iceberg handle compliance requirements. - [Securing Apache Iceberg Tables with Fine-Grained Row and Column Level Access Control](https://iceberglakehouse.com/posts/iceberg-row-column-access-control/) (2026-05-28): Apache Iceberg doesn't enforce row and column security on its own. Learn how catalog-level governance with Apache Polaris and Dremio delivers enterprise FGAC. - [Designing an Immutable Data Lakehouse: Best Practices for Iceberg Snapshot Expiration](https://iceberglakehouse.com/posts/iceberg-snapshot-expiration/) (2026-05-28): Iceberg snapshot bloat silently kills query performance. Learn the best practices for snapshot expiration, manifest cleanup, and automated table maintenance. - [Decoupling Storage and Compute in Apache Iceberg: A Deep Dive into Cost Optimization](https://iceberglakehouse.com/posts/iceberg-storage-compute-decoupling/) (2026-05-28): Learn how Apache Iceberg decouples storage from compute to cut data platform costs, enable multi-engine routing, and eliminate proprietary lock-in. - [The Endgame – Building an Autonomous Optimization Pipeline for Apache Iceberg](https://iceberglakehouse.com/posts/iceberg-autonomous-optimization-pipeline/) (2025-09-16): Learn how to automate compaction, snapshot expiration, and layout optimization in Apache Iceberg using metadata-driven triggers and orchestration tools. - [Managing Large-Scale Optimizations – Parallelism, Checkpointing, and Fail Recovery](https://iceberglakehouse.com/posts/iceberg-large-scale-optimization/) (2025-09-09): Learn how to scale Apache Iceberg table optimizations across large datasets using parallelism, checkpointing, and fail recovery to ensure reliability. - [Hidden Pitfalls – Compaction and Partition Evolution in Apache Iceberg](https://iceberglakehouse.com/posts/iceberg-partition-evolution-compaction/) (2025-09-02): Partition evolution in Apache Iceberg is a powerful feature, but if not managed carefully, it can introduce fragmentation and impact compaction. - [Using Iceberg Metadata Tables to Determine When Compaction Is Needed](https://iceberglakehouse.com/posts/iceberg-metadata-triggered-compaction/) (2025-08-26): Discover how to use Apache Iceberg's metadata tables to proactively detect small files, bloated manifests, and table fragmentation - so you can trigger. - [Designing the Ideal Cadence for Compaction and Snapshot Expiration](https://iceberglakehouse.com/posts/iceberg-optimization-cadence/) (2025-08-19): Learn how to design an effective schedule for compaction and snapshot expiration in Apache Iceberg to balance cost, performance, and data freshness. - [Avoiding Metadata Bloat with Snapshot Expiration and Rewriting Manifests](https://iceberglakehouse.com/posts/iceberg-metadata-bloat-cleanup/) (2025-08-12): Learn how to prevent and clean up metadata bloat in Apache Iceberg by expiring snapshots and rewriting manifests for better performance and manageability. - [Smarter Data Layout – Sorting and Clustering Iceberg Tables](https://iceberglakehouse.com/posts/iceberg-clustering-sorting-zorder/) (2025-08-05): Improve query performance in Apache Iceberg by organizing your data layout with sorting and Z-order clustering. - [Optimizing Compaction for Streaming Workloads in Apache Iceberg](https://iceberglakehouse.com/posts/iceberg-streaming-compaction/) (2025-07-29): Learn how to design fast, incremental compaction strategies in Apache Iceberg to support high-throughput streaming pipelines without disrupting freshness. - [The Basics of Compaction – Bin Packing Your Data for Efficiency](https://iceberglakehouse.com/posts/iceberg-optimization-compaction-basics/) (2025-07-22): Learn how standard compaction works in Apache Iceberg and why bin packing your data files is essential for maintaining query performance and cost efficiency. - [The Cost of Neglect – How Apache Iceberg Tables Degrade Without Optimization](https://iceberglakehouse.com/posts/iceberg-optimization-degradation/) (2025-07-15): Learn how Apache Iceberg tables can degrade over time without optimization and what issues this causes for performance, cost, and governance. ## Articles: Data Lakehouse Architecture - [The Open Lakehouse Explained, Then Built on Your Laptop with Dremio and MinIO](https://iceberglakehouse.com/posts/open-lakehouse-on-your-laptop/) (2026-09-10): The five layers of the open lakehouse explained, then a lab: Parquet, Iceberg, Polaris, Arrow, and Ossie running in two containers on your own machine. - [Postgres Meets the Lakehouse: pg_lake, pg_duckdb, and When Postgres Is Enough](https://iceberglakehouse.com/posts/postgres-meets-the-lakehouse/) (2026-09-02): What pg_lake, pg_duckdb, and pg_mooncake do at the Iceberg level, and honest thresholds for when Postgres is enough. - [Building Apache Iceberg Lakehouses That Run Without an Internet Connection](https://iceberglakehouse.com/posts/building-air-gapped-iceberg-lakehouse/) (2026-08-04): How to build an Apache Iceberg lakehouse that runs fully offline: storage, catalog, compute, cross-zone transfer, compliance, and the failure modes that bite. - [Building an Honest TCO Model for Open Lakehouses and Proprietary Warehouses](https://iceberglakehouse.com/posts/lakehouse-vs-warehouse-tco/) (2026-08-04): An honest TCO framework for open lakehouses versus proprietary warehouses: five cost categories, measured numbers, sensitivity analysis, and where each. - [A Migration Playbook for Moving Legacy Warehouses onto Apache Iceberg](https://iceberglakehouse.com/posts/warehouse-to-iceberg-migration-playbook/) (2026-08-04): A dependency-first playbook for migrating legacy warehouses onto Apache Iceberg: snapshot vs migrate vs add_files, four-level parity validation. - [Automated Materialized Views in Dremio](https://iceberglakehouse.com/posts/automated-materialized-views-autonomous-table-optimization-dremio-agentic-lakehouse/) (2026-07-13): An in-depth exploration of automated materialized views in dremio - [Data Lakehouse Open Standards for AI Agents](https://iceberglakehouse.com/posts/data-lakehouse-open-standards-agentic-era/) (2026-07-13): An in-depth exploration of data lakehouse open standards for ai agents - [Dremio Lakehouse AI Report: Agentic Lessons](https://iceberglakehouse.com/posts/dremio-state-data-lakehouse-ai-report-agentic-lakehouse/) (2026-07-13): An in-depth exploration of dremio lakehouse ai report: agentic lessons - [Five-Layer Agentic Lakehouse Architecture](https://iceberglakehouse.com/posts/five-layer-agentic-lakehouse-architecture-enterprise-pattern/) (2026-07-13): An in-depth exploration of five-layer agentic lakehouse architecture - [Hybrid Lakehouse Design for Regulated Markets](https://iceberglakehouse.com/posts/hybrid-lakehouse-regulated-markets-on-prem-public-metadata/) (2026-07-13): An in-depth exploration of hybrid lakehouse design for regulated markets - [Zero-Copy Mirroring to Open Iceberg Tables](https://iceberglakehouse.com/posts/zero-copy-mirroring-migration-proprietary-warehouses-open-iceberg/) (2026-07-13): An in-depth exploration of zero-copy mirroring to open iceberg tables - [Federation and the Lakehouse: Two Roads to Unified Data Access, and How to Know Which One to Take](https://iceberglakehouse.com/posts/federation-and-lakehouse/) (2026-07-06): Every data strategy document written this decade contains some version of the same sentence: we need a single place to access all our data. The sen... - [The Who, What, and Why of Semantic Layers: The Layer That Decides Whether Your Numbers Can Be Trusted](https://iceberglakehouse.com/posts/who-what-why-semantic-layers/) (2026-07-06): There is a survey statistic making the rounds this year that I cannot stop quoting: 84 percent of data teams report regularly encountering conflict... - [AI-Ready Metadata Prevents Query Failures](https://iceberglakehouse.com/posts/ai-ready-metadata-lineage-quality-llm-query-failures/) (2026-06-22): AI-ready metadata reduces query failures by making ownership, freshness, lineage, quality, and policy visible at execution time. - [Autonomous Materialization for Agentic Analytics](https://iceberglakehouse.com/posts/autonomous-materialization-ai-agents-table-performance/) (2026-06-22): Autonomous materialization is useful when it is tied to workload evidence, governance checks, and lifecycle management. - [Composable Semantic Layers for Analytical Agents](https://iceberglakehouse.com/posts/beyond-metrics-lists-composable-semantic-layers-agents/) (2026-06-22): AI agents need more than metric names. They need composable business logic that survives multi-step analysis. - [Built for Agents and Managed by Agents](https://iceberglakehouse.com/posts/built-for-agents-managed-by-agents-dremio-agentic-lakehouse/) (2026-06-22): Dremio Agentic Lakehouse is easiest to understand as two ideas: data built for agent access and platform work managed by agents. - [ClickHouse in the Loop for Active Agents](https://iceberglakehouse.com/posts/clickhouse-loop-real-time-event-streams-active-agents/) (2026-06-22): Low-latency analytical systems can help active agents, but only when event loops include validation, context, and safety boundaries. - [The Context Layer for AI Agents](https://iceberglakehouse.com/posts/context-layer-semantic-lineage-quality-ai-agents/) (2026-06-22): A semantic layer is necessary, but agents also need lineage, quality, freshness, compliance, and ownership context. - [Lakehouse as the Operating Layer for Agentic AI](https://iceberglakehouse.com/posts/databricks-summit-2026-lakehouse-agentic-ai-operating-layer/) (2026-06-22): Agentic AI announcements are useful when they validate the need for governed data, semantic context, and cost-aware execution. - [Event-Driven Table Compaction with Agents](https://iceberglakehouse.com/posts/event-driven-table-compaction-agentic-coordination/) (2026-06-22): Event-driven compaction is valuable when agents coordinate maintenance with workload signals, table health, and commit safety. - [Fabric Agentic Analytics and Lakehouse Schema Design](https://iceberglakehouse.com/posts/fabric-build-2026-lakehouse-schema-agentic-analytics/) (2026-06-22): Microsoft Fabric agentic analytics is a reminder that schemas, semantic models, and governed lakehouse design now shape AI behavior. - [Iceberg v4 Performance: Root Manifests and Calls](https://iceberglakehouse.com/posts/iceberg-v4-performance-root-manifests-combined-calls/) (2026-06-22): Apache Iceberg v4 discussion should focus on planning cost, metadata layout, and object storage round trips, not vague claims about faster tables. - [What Is LTAP in the Lakehouse?](https://iceberglakehouse.com/posts/ltap-lakehouse-transactional-analytical-processing-2026/) (2026-06-22): Lakehouse transactional analytical processing is useful only when teams define freshness, isolation, and workload boundaries clearly. - [The Model Is Not the Moat](https://iceberglakehouse.com/posts/model-not-moat-semantic-lakehouse-layer-enterprise-ai/) (2026-06-22): Enterprise AI advantage increasingly comes from governed context, semantic models, and operational data contracts, not only from model choice. - [PyIceberg at Scale Without Apache Spark](https://iceberglakehouse.com/posts/pyiceberg-scale-high-concurrency-no-spark/) (2026-06-22): Python-first Iceberg work is useful when it stays honest about what Python should and should not do. - [The Real-Time Lakehouse with Streaming and Iceberg](https://iceberglakehouse.com/posts/real-time-lakehouse-streaming-sql-cold-iceberg-storage/) (2026-06-22): The real-time lakehouse is not one engine. It is a contract between streams, table commits, query paths, and freshness expectations. - [REST Catalog V2 LoadTable and Client Capability](https://iceberglakehouse.com/posts/rest-catalog-v2-loadtable-protocol-design/) (2026-06-22): REST Catalog V2 LoadTable work matters because clients and catalogs need explicit contracts, not optimistic assumptions. - [Rust vs C++ in Native Iceberg Scan Operators](https://iceberglakehouse.com/posts/rust-vs-cpp-table-layer-native-scan-operators/) (2026-06-22): The Rust versus C++ discussion is really about table-layer execution safety, interoperability, and performance envelopes. - [Server-Side Commit Deconflicting in REST Catalogs](https://iceberglakehouse.com/posts/server-side-commit-deconflicting-rest-catalogs/) (2026-06-22): Server-side commit deconflicting is about moving concurrency control closer to the catalog contract. - [Snowflake Interoperable Lakehouse Lessons](https://iceberglakehouse.com/posts/snowflake-interoperable-lakehouse-production-lessons/) (2026-06-22): Interoperable lakehouse announcements matter when they change production contracts, not just import and export narratives. - [Unity AI Gateway vs Apache Polaris Control Planes](https://iceberglakehouse.com/posts/unity-ai-gateway-vs-apache-polaris-governance-control-plane/) (2026-06-22): The right comparison is not vendor scoreboard. It is closed AI governance gateway versus open catalog control plane. - [Evaluating the TCO of an Open Lakehouse vs. Proprietary Data Warehouses](https://iceberglakehouse.com/posts/open-lakehouse-vs-proprietary-warehouse-tco/) (2026-05-28): Open lakehouse vs proprietary warehouse: a comprehensive TCO breakdown covering storage, compute, engineering, and hidden costs to help you make the right. - [The Data Lakehouse - The Benefits and Enhancing Implementation](https://iceberglakehouse.com/posts/2025-01-the-data-lakehouse-benefits-and-enhancing/) (2025-01-31): Understanding the value of a lakehouse and how to get that value faster - [When to use Apache Xtable or Delta Lake Uniform for Data Lakehouse Interoperability](https://iceberglakehouse.com/posts/2025-01-xtable-or-uniform/) (2025-01-07): A Guide on when to use Apache Xtable or Delta Lake Uniform for Data Lakehouse Interoperability - [Deep Dive into Dremio's File-based Auto Ingestion into Apache Iceberg Tables](https://iceberglakehouse.com/posts/2024-11-deep-dive-auto-ingest-dremio-iceberg/) (2024-11-15): Auto ingesting data from JSON, CSV, and Parquet files into Apache Iceberg Tables - [Dremio, Apache Iceberg and their role in AI-Ready Data](https://iceberglakehouse.com/posts/2024-11-dremio-and-ai-ready-data/) (2024-11-05): The Role of Dremio and Apache Iceberg in AI-Ready Data - [Hands-on with Apache Iceberg & Dremio on Your Laptop within 10 Minutes](https://iceberglakehouse.com/posts/2024-10-hands-on-with-iceberg-dremio-laptop/) (2024-10-31): How to get hands-on with Apache Iceberg - [All About Parquet Part 01 - An Introduction](https://iceberglakehouse.com/posts/2024-10-all-about-parquet-part-01/) (2024-10-21): All about the Apache Parquet File Format - [All About Parquet Part 02 - Parquet's Columnar Storage Model](https://iceberglakehouse.com/posts/2024-10-all-about-parquet-part-02/) (2024-10-21): All about the Apache Parquet File Format - [All About Parquet Part 03 - Parquet File Structure | Pages, Row Groups, and Columns](https://iceberglakehouse.com/posts/2024-10-all-about-parquet-part-03/) (2024-10-21): All about the Apache Parquet File Format - [All About Parquet Part 04 - Schema Evolution in Parquet](https://iceberglakehouse.com/posts/2024-10-all-about-parquet-part-04/) (2024-10-21): All about the Apache Parquet File Format - [All About Parquet Part 05 - Compression Techniques in Parquet](https://iceberglakehouse.com/posts/2024-10-all-about-parquet-part-05/) (2024-10-21): All about the Apache Parquet File Format - [All About Parquet Part 06 - Encoding in Parquet | Optimizing for Storage](https://iceberglakehouse.com/posts/2024-10-all-about-parquet-part-06/) (2024-10-21): All about the Apache Parquet File Format - [All About Parquet Part 07 - Metadata in Parquet | Improving Data Efficiency](https://iceberglakehouse.com/posts/2024-10-all-about-parquet-part-07/) (2024-10-21): All about the Apache Parquet File Format - [All About Parquet Part 08 - Reading and Writing Parquet Files in Python](https://iceberglakehouse.com/posts/2024-10-all-about-parquet-part-08/) (2024-10-21): All about the Apache Parquet File Format - [All About Parquet Part 09 - Parquet in Data Lake Architectures](https://iceberglakehouse.com/posts/2024-10-all-about-parquet-part-09/) (2024-10-21): All about the Apache Parquet File Format - [All About Parquet Part 10 - Performance Tuning and Best Practices with Parquet](https://iceberglakehouse.com/posts/2024-10-all-about-parquet-part-10/) (2024-10-21): All about the Apache Parquet File Format - [A Guide to dbt Macros - Purpose, Benefits, and Usage](https://iceberglakehouse.com/posts/2024-10-a-guide-to-dbt-macros/) (2024-10-18): Learning about dbt Macros - [Getting Started with Data Analytics Using PyArrow in Python](https://iceberglakehouse.com/posts/2024-10-getting-started-with-pyarrow-in-python/) (2024-10-15): Learning to work with PyArrow to run analytics - [What is Three-Tier Data (Bronze, Silver, Gold) and How Dremio Simplifies It](https://iceberglakehouse.com/posts/2024-10-bronze-silver-gold-data/) (2024-10-09): Process Data from Raw to Clean Aggregated Data - [A Brief Guide to the Governance of Apache Iceberg Tables](https://iceberglakehouse.com/posts/2024-10-governing-apache-iceberg-tables/) (2024-10-07): Controlling Access to your Apache Iceberg Tables - [Exploring Data Operations with PySpark, Pandas, DuckDB, Polars, and DataFusion in a Python Notebook](https://iceberglakehouse.com/posts/2024-10-exploring-data-operations-python/) (2024-10-07): Learning to work with Python to ingest and query data - [Ultimate Directory of Apache Iceberg Resources](https://iceberglakehouse.com/posts/2024-10-ultimate-directory-of-apache-iceberg-resources/) (2024-10-05): Apache Iceberg Education, Tutorials and more! - [Change Data Capture (CDC) when there is no CDC](https://iceberglakehouse.com/posts/2024-10-cdc-when-there-is-no-cdc/) (2024-10-04): Handling Synching Changing Data Across Systems - [Virtualization + Lakehouse + Mesh = Data At Scale](https://iceberglakehouse.com/posts/2024-9-virtualization-lakehouse-mesh-data-at-scale/) (2024-09-25): Combining Centralization and Decentralization for Data at Scale - [Why Data Analysts, Engineers, Architects and Scientists Should Care about Dremio and Apache Iceberg](https://iceberglakehouse.com/posts/2024-9-why-dremio-iceberg-matters/) (2024-09-10): The Evolving Data Lakehouse World - [Hands-on with Apache Iceberg on Your Laptop - Deep Dive with Apache Spark, Nessie, Minio, Dremio, Polars and Seaborn](https://iceberglakehouse.com/posts/2024-9-hands-on-iceberg-dremio-minio-nessie/) (2024-09-10): The Evolving Data Lakehouse World - [5 Trends in the Data Lakehouse Space](https://iceberglakehouse.com/posts/2024-9-five-trends-in-data-lakehouse/) (2024-09-01): The Evolving Data Lakehouse World - [Using the alexmerced/datanotebook Docker Image](https://iceberglakehouse.com/posts/2024-8-using-the-alexmerced-datanotebook-image/) (2024-08-30): Setting up a quick and easy data environment for data science and analytics - [Understanding Apache Iceberg Delete Files](https://iceberglakehouse.com/posts/2024-8-understanding-apache-iceberg-delete-files/) (2024-08-29): Continuing the Understand Apache Iceberg series, this article delves into the Manifest, a critical component of Apache Iceberg's architecture. - [Understanding the Apache Iceberg Manifest](https://iceberglakehouse.com/posts/2024-8-understanding-apache-iceberg-manifest/) (2024-08-27): Continuing the Understand Apache Iceberg series, this article delves into the Manifest, a critical component of Apache Iceberg's architecture. - [Understanding the Apache Iceberg Manifest List (Snapshot)](https://iceberglakehouse.com/posts/2024-8-understanding-apache-iceberg-manifest-list/) (2024-08-25): Continuing the Understand Apache Iceberg series, this article delves into the Manifest List, a critical component of Apache Iceberg's architecture. - [Understanding Apache Iceberg's Metadata.json](https://iceberglakehouse.com/posts/2024-8-apache-iceberg-metadata-json/) (2024-08-21): The role and content of the metadata.json - [What Apache Iceberg REST Catalog is and isn't](https://iceberglakehouse.com/posts/2024-8-what-apache-iceberg-rest-catalog-is-and-isnt/) (2024-08-18): The Who, What and Why of Data Lakehouses - [ACID Guarantees and Apache Iceberg - Turning Any Storage into a Data Warehouse](https://iceberglakehouse.com/posts/2024-8-acid-guarantees-and-apache-iceberg/) (2024-08-15): What are ACID Guarantees? WHy do they matter? - [Data Lakehouse 101 - The Who, What and Why of Data Lakehouses](https://iceberglakehouse.com/posts/2024-8-data-lakehouses-101/) (2024-08-05): The Who, What and Why of Data Lakehouses - [Apache Iceberg Reliability](https://iceberglakehouse.com/posts/2024-7-apache-iceberg-reliability/) (2024-07-26): Why Apache Iceberg Works - [Understanding the Polaris Iceberg Catalog and Its Architecture](https://iceberglakehouse.com/posts/2024-7-understanding-polaris-apache-iceberg-catalog/) (2024-07-20): Learn about the new open source Iceberg Catalog in Town - [Databases Deconstructed - The Value of Data Lakehouses and Table Formats](https://iceberglakehouse.com/posts/2024-7-databases-decontstructed-value-of-data-lakehouses-and-table-formats/) (2024-07-12): Building up the Data Lakehouse - [Partitioning with Apache Iceberg - A Deep Dive](https://iceberglakehouse.com/posts/2024-5-partitioning-with-apache-iceberg-deep-dive/) (2024-05-29): Benefits of Apache Iceberg Partition Evolution and Hidden Partitioning - [3 Reasons Data Engineers Should Embrace Apache Iceberg](https://iceberglakehouse.com/posts/2024-5-3-reasons-data-engineers-should-embrace-apache-iceberg/) (2024-05-15): Benefits of Apache Iceberg - [Understanding the Future of Apache Iceberg Catalogs](https://iceberglakehouse.com/posts/2024-4-understanding-the-future-of-apache-iceberg-catalogs/) (2024-04-04): Java, Rest and the expanding open lakehouse ecosystem - [End-to-End Basic Data Engineering Tutorial (Spark, Dremio, Superset)](https://iceberglakehouse.com/posts/2024-4-end-to-end-data-engineering-tutorial-spark-dremio-superset/) (2024-04-01): Ingesting Data and Building BI Dashboards - [5 Reasons Dremio is the Ideal Apache Iceberg Lakehouse Platform](https://iceberglakehouse.com/posts/2024-3-5-reasons-dremio-is-the-ideal-iceberg-lakehouse-platform/) (2024-03-09): Understanding how catalogs work and which one to choose - [The Apache Iceberg Lakehouse - The Great Data Equalizer](https://iceberglakehouse.com/posts/2024-3-06-apache-iceberg-the-great-data-equalizer/) (2024-03-06): Disrupting the Snowflake/Databricks status quo - [10 Reasons to Make Apache Iceberg and Dremio Part of Your Data Lakehouse Strategy](https://iceberglakehouse.com/posts/2024-3-10-reasons-to-make-dremio-part-of-your-data-lakehouse-strategy/) (2024-03-01): Understanding how catalogs work and which one to choose - [A deep dive into the concept and world of Apache Iceberg Catalogs](https://iceberglakehouse.com/posts/2024-3-deep-dive-into-apache-iceberg-catalogs/) (2024-03-01): Understanding how catalogs work and which one to choose - [What is the Data Lakehouse and the Role of Apache Iceberg, Nessie and Dremio?](https://iceberglakehouse.com/posts/2024-2-what-a-data-lakehouse-and-role-of-dremio-iceberg-nessie/) (2024-02-21): Understanding the Value of the Data Lakehouse - [Partitioning Practices in Apache Hive and Apache Iceberg](https://iceberglakehouse.com/posts/2024-2-partitioning-in-apache-hive-and-apache-iceberg/) (2024-02-12): Deep Dive in Data Lake Table Partitioning - [Embracing the Future of Data Management - Why Choose Lakehouse, Iceberg, and Dremio?](https://iceberglakehouse.com/posts/2024-1-why-choose-lakehouse-iceberg-dremio/) (2024-01-25): The Future of Data Platforms - [Open Lakehouse Engineering/Apache Iceberg Lakehouse Engineering - A Directory of Resources](https://iceberglakehouse.com/posts/2024-1-open-lakehouse-engineering-resources/) (2024-01-19): Resources for learning how to Engineer an Open Data Lakehouse - [Nessie - An Alternative to Hive & JDBC for Self-Managed Apache Iceberg Catalogs](https://iceberglakehouse.com/posts/2024-1-nessie-an-alternative-to-hive/) (2024-01-08): Nessie is the only open-source catalog implementation specifically for Apache Iceberg. - [Apache Iceberg, Git-Like Catalog Versioning and Data Lakehouse Management - Pillars of a Robust Data Lakehouse Platform](https://iceberglakehouse.com/posts/2024-1-apache-iceberg-git-life-catalog-versioning/) (2024-01-03): This is where the combined power of Dremio’s Lakehouse Management features and Project Nessie's catalog-level versioning comes into play. ## Articles: Agentic Analytics and Semantic Layers - [How to Build a Golden-Question Benchmark for Your Data Agents](https://iceberglakehouse.com/posts/data-agent-golden-question-benchmark/) (2026-10-05): Build a golden-question benchmark for data agents: frozen Iceberg tags, execution-based scoring with Arrow, and CI gates. - [When a Data Agent Gives a Wrong Number: Incident Response for Agentic Analytics](https://iceberglakehouse.com/posts/data-agent-wrong-number-incident-response/) (2026-10-05): A plan for when a data agent gives a wrong number: severity levels, containment, Iceberg time travel reproduction, and blast radius queries. - [Multi-Tenant Data Agents: Serving Many Customers From One Lakehouse](https://iceberglakehouse.com/posts/multi-tenant-data-agents-lakehouse/) (2026-10-05): Serve many customers from one lakehouse without leaking between them: isolation models, Polaris grants, and tenant-scoped agent state. - [Release Management for Data Agents: Shipping Prompts, Tools, Models, and Metrics Without Breaking Answers](https://iceberglakehouse.com/posts/release-management-data-agents/) (2026-10-05): Ship prompts, tools, models, and metric changes without breaking answers: agent manifests, shadow mode, canaries, and rollback. - [Semantic Caching for Data Agents Without Serving Stale or Leaked Numbers](https://iceberglakehouse.com/posts/semantic-caching-data-agents/) (2026-10-05): Cache data agent results safely with canonical query keys, Iceberg snapshot freshness checks, and permission-aware cache keys. - [How Apache Ossie Is Deciding What Agents and BI Tools Can Ask a Semantic Layer](https://iceberglakehouse.com/posts/ossie-semantic-query-interface-agents/) (2026-09-21): How Apache Ossie's layered query design gives AI agents both a constrained dimensional interface and a grain-safe SQL interface for semantic layers. - [Apache Ossie and Apache Polaris: Putting Semantic Models in the Open Catalog](https://iceberglakehouse.com/posts/apache-ossie-polaris-semantic-models/) (2026-08-25): Apache Ossie and Polaris put metric definitions in the open catalog. What the spec covers, what Polaris stores, and what is still unfinished. - [Semantic Layer Federation: One Logical Model Over Data on Three Clouds](https://iceberglakehouse.com/posts/semantic-layer-federation-three-clouds/) (2026-08-25): One logical model over Iceberg and databases on three clouds. Pushdown, egress, Reflections, and where semantic federation still breaks. - [GSA MCP Servers for Open Data AI Agents](https://iceberglakehouse.com/posts/gsa-model-context-protocol-server-hackathon-open-data-ai-agents/) (2026-07-13): An in-depth exploration of gsa mcp servers for open data ai agents - [MCP Stateless Gateways for Data Agents](https://iceberglakehouse.com/posts/mcp-stateless-specifications-scalable-data-gateways-ai-agents/) (2026-07-13): An in-depth exploration of mcp stateless gateways for data agents - [The Buyer's Scorecard for Agentic Analytics: Evaluating Tooling in the Enterprise AI Era](https://iceberglakehouse.com/posts/buyers-scorecard-agentic-analytics-enterprise-ai-era/) (2026-07-06): Agentic analytics demos are easy to enjoy and hard to evaluate. A user asks a question, an assistant answers, a chart appears, and the room leans f... - [Designing Idempotent Pipelines in the Agentic Lakehouse: Eliminating Double-Write Anomalies](https://iceberglakehouse.com/posts/idempotent-pipelines-agentic-lakehouse-double-write-anomalies/) (2026-07-06): Agents retry. Networks fail. Jobs time out after doing some work. APIs return ambiguous responses. Schedulers run the same workflow twice. A human... - [The Metric Contract Mandate: Standardizing Semantic Layers Before AI Agent Access](https://iceberglakehouse.com/posts/metric-contract-mandate-semantic-layers-ai-agent-access/) (2026-07-06): AI agents are very good at moving quickly. That is the opportunity and the risk. If an agent can inspect metadata, generate queries, compare result... - [Trustworthy Concurrency in the Agentic Lakehouse: Reconciling Academic Proofs with High-Frequency Production Writes](https://iceberglakehouse.com/posts/trustworthy-concurrency-agentic-lakehouse-production-writes/) (2026-07-06): Agentic lakehouses change the concurrency conversation. Traditional data pipelines already deal with overlapping jobs, retries, compaction, merges... ## Articles: AI and Agents - [Small Language Models for Data Agents: Where They Win and Where They Fail](https://iceberglakehouse.com/posts/small-language-models-data-agents/) (2026-10-05): Where small language models win inside a data agent and where they fail, plus a routing architecture that escalates only when needed. - [Turning an Analytics Question Into a Verified Agentic Graph](https://iceberglakehouse.com/posts/ags-multi-agent-analytics-workflow/) (2026-09-28): A complete AGS 1.0 graph for governed metric questions, with verification gates, deterministic check scripts, and reconciliation against a semantic layer. - [Twelve Queries That Pass Every Schema Check and Still Return the Wrong Number](https://iceberglakehouse.com/posts/semantic-failure-catalog-ai-agent-queries/) (2026-09-28): Twelve query patterns that pass schema validation and still return wrong numbers, with detectors, required metadata, and tested code for each. - [Fast Classification Models, LLMs, and the Apache Iceberg Lakehouse](https://iceberglakehouse.com/posts/jev-classification-models-iceberg-lakehouse/) (2026-09-21): How fast classification models like Jev alongside open alternatives such as GLiClass compare with LLMs, and how to run both together inside an Apache Iceberg lakehouse. - [Agentic Data Architecture](https://iceberglakehouse.com/posts/agentic-data-architecture/) (2026-09-10): A six-layer reference architecture for agents on company data: planners, tool boundaries, identity, the semantic layer, and what breaks when a layer is missing. - [Context Engineering for Data Agents](https://iceberglakehouse.com/posts/context-engineering-for-data-agents/) (2026-09-10): Why text-to-SQL accuracy collapses on enterprise schemas, the five kinds of context an agent needs, where each one hides, and how to make the semantics legible. - [The Data Team of the Agentic Era: Generalists Owning End-to-End Workflows](https://iceberglakehouse.com/posts/data-team-of-the-agentic-era/) (2026-09-02): The case for generalists owning end-to-end data workflows with agents, the counterargument, and how to make the transition work. - [Will AI Replace Data Engineers?](https://iceberglakehouse.com/posts/will-ai-replace-data-engineers/) (2026-09-02): What the evidence shows about whether AI replaces data engineers, which parts of the job compress, and which parts do not. - [Securing the Agentic Lakehouse Gateway: Preventing Prompt Injection and Data Exfiltration](https://iceberglakehouse.com/posts/agentic-lakehouse-gateway-prompt-injection/) (2026-08-25): Agentic lakehouse gateways face prompt injection and exfiltration through query results. A threat model and defenses for the layer in front of data. - [Metric Contracts in Code: Testing, Versioning, and Serving Business Logic to Multi-Agent Systems](https://iceberglakehouse.com/posts/metric-contracts-code-multi-agent/) (2026-08-25): Metric contracts in code let teams test, version, and serve business logic to multi-agent systems without each agent inventing its own SQL. - [The Agent Is Now a Named Coworker, and It Needs a File Format](https://iceberglakehouse.com/posts/agents-with-personalities/) (2026-08-24): Named, persistent agents need a file format. Open Agent Profile, Buzz, Grok Bot, and Hermes Bot Mode show why a portable agent identity matters. - [Your Agent Should Answer the Phone: A Field Guide to AI Gateways on Slack, Discord, Telegram, Signal, and Teams](https://iceberglakehouse.com/posts/ai-gateways-field-guide/) (2026-08-24): A field guide to AI gateways on Slack, Discord, Telegram, Signal, and Teams: architecture, auth, cost, and the failure modes that matter. - [Graphs in AI Engineering Have Solved Three Problems. The Fourth Is the Plan.](https://iceberglakehouse.com/posts/graphs-in-ai-engineering/) (2026-08-24): Knowledge graphs, GraphRAG, and LangGraph solved three problems. The fourth is the work itself: a reviewable graph of bounded agentic loops. - [The Five Layers of an Agentic Lakehouse](https://iceberglakehouse.com/posts/five-layers-agentic-lakehouse/) (2026-08-19): The five layers of an agentic lakehouse: Storage, Catalog, Semantic, Gateway, and Agent Surface, and how one question travels through all of them. - [Goal-Directed Data Quality Agents: Anomaly Quarantine on Apache Iceberg](https://iceberglakehouse.com/posts/goal-directed-data-quality-agents-iceberg/) (2026-08-19): Goal-directed data quality agents that watch Apache Iceberg tables, detect anomalies, and quarantine suspect data safely with snapshot isolation and branches. - [Managing the TCO of Agentic Analytics: Token Budgets, Query Throttles, and the Economics of Autonomy](https://iceberglakehouse.com/posts/managing-tco-agentic-analytics/) (2026-08-19): Managing the total cost of agentic analytics: token budgets, query throttles, unit economics, and the FinOps discipline that keeps AI spend under control. - [Metric Contracts in 2026: Standardizing Business Logic Across Multi-Agent Frameworks](https://iceberglakehouse.com/posts/metric-contracts-2026/) (2026-08-19): Metric contracts in 2026: versioned, testable definitions of business logic that let multi-agent frameworks compute revenue identically, with OSI interchange. - [Query Routing at Machine Scale: Multi-Engine Workload Distribution for the Agentic Lakehouse](https://iceberglakehouse.com/posts/query-routing-machine-scale/) (2026-08-19): Query routing at machine scale for the agentic lakehouse: engine selection, acceleration substitution, admission control, and placement. - [Building Stateless AI Tool Gateways with FastMCP, the 2026 MCP Spec, and Kubernetes](https://iceberglakehouse.com/posts/stateless-mcp-tool-gateways-fastmcp/) (2026-08-19): How to build stateless AI tool gateways with FastMCP, the 2026 MCP specification, and Kubernetes, and why statelessness finally makes MCP scale. - [The Plan and the Worker: Two Open Specifications for Agent Harnesses](https://iceberglakehouse.com/posts/agentic-graph-open-agent-profile-two-open-specs-agent-harnesses/) (2026-08-10): Two open specifications, the Agentic Graph Specification and the Open Agent Profile, turn agent plans and agent identity into portable, reviewable files. - [Budgeting for Agentic Analytics When Every Question Costs Something Different](https://iceberglakehouse.com/posts/agentic-analytics-tco-token-budgets/) (2026-08-04): Budgeting for agentic analytics when every question costs something different: token economics, query economics, instrumentation, and the cost controls. - [The Five Layers of an Agentic Lakehouse and Where the MCP Server Sits](https://iceberglakehouse.com/posts/agentic-lakehouse-mcp-architecture/) (2026-08-04): The five layers of an agentic lakehouse and where the MCP server sits: storage, catalog, semantic layer, MCP gateway, and agent surface, plus identity. - [Wiring Analytical Queries to Transactional APIs in Closed-Loop Decision Agents](https://iceberglakehouse.com/posts/closed-loop-decision-agents/) (2026-08-04): Wiring analytical queries to transactional APIs in closed-loop decision agents: conditional writes, sagas with compensations, decision records, and blast. - [Why Agentic AI Needs a Governed Semantic Layer Behind the Model Context Protocol](https://iceberglakehouse.com/posts/mcp-governed-semantic-layer/) (2026-08-04): Why agentic AI needs a governed semantic layer behind the Model Context Protocol: metric consistency, access control, Apache Ossie for portable. - [Moving From Supply Chain Dashboards to Decision Loops With the Model Context Protocol](https://iceberglakehouse.com/posts/mcp-supply-chain-decision-loops/) (2026-08-04): Moving from supply chain dashboards to decision loops with MCP: sense, decide, act, and verify, with typed action tools, idempotency keys, and graduated. - [Metric Contracts as the Interface AI Agents Actually Need](https://iceberglakehouse.com/posts/metric-contracts-for-ai-agents/) (2026-08-04): Metric contracts as the interface AI agents need: calculation, inclusion rules, grain, temporal semantics, ownership, semantic versioning, and testing. - [Designing Policy-Aware Telemetry Tables for AI Systems in Apache Iceberg](https://iceberglakehouse.com/posts/policy-aware-ai-telemetry-iceberg/) (2026-08-04): Designing policy-aware AI telemetry tables in Apache Iceberg: what to log, tamper evidence, retention against conflicting deletion requirements. - [Defending the Lakehouse Gateway Against Prompt Injection and Data Exfiltration](https://iceberglakehouse.com/posts/preventing-prompt-injection-lakehouse-gateway/) (2026-08-04): Defending the lakehouse gateway against prompt injection and data exfiltration: per-user identity, no-SQL tool surfaces, volume bounds, and detection. - [Guardrails for Analytics Agents That Do More Than Answer Questions](https://iceberglakehouse.com/posts/agent-guardrails-human-in-the-loop/) (2026-07-28): The risk isn't agents going rogue, it's agents acting correctly on bad input at machine speed. - [Building Agent Telemetry Tables in Iceberg That Survive an Audit](https://iceberglakehouse.com/posts/agent-telemetry-iceberg-audit/) (2026-07-28): A practical guide to building agent decision traces in Apache Iceberg that support audit reconstruction, governance review, and cost attribution. - [What Agentic Analytics Actually Costs, and How to Keep It Bounded](https://iceberglakehouse.com/posts/agentic-analytics-tco/) (2026-07-28): Agent analytics generates two cost streams that scale on different variables. Here's the arithmetic, the levers that actually move the number, and how. - [The Five Layers Between Your Lakehouse and a Trustworthy Agent](https://iceberglakehouse.com/posts/five-layer-agentic-lakehouse/) (2026-07-28): Agent reliability is a property of the stack the model sits on. Five layers with distinct owners and failure modes turn the agent is unreliable. - [Governing What Agents Cost You](https://iceberglakehouse.com/posts/agent-cost-governance/) (2026-07-25): Agents break the four assumptions analytics platforms were built on. A practical guide to identity, budgets, semantic layers, caching, and instrumentation. - [The Model Class Enterprise Data Teams Are Not Using Yet](https://iceberglakehouse.com/posts/tabular-foundation-models/) (2026-07-25): Tabular foundation models perform supervised learning in a single forward pass with no training run. - [The Own Goal: What the Hugging Face Breach Reveals About Why Open Models Are a Security Necessity](https://iceberglakehouse.com/posts/value-of-open-models-security/) (2026-07-24): How the 2026 Hugging Face breach proved that open models are a security necessity, closed models failed to defend, an open Chinese model succeeded. - [Active Analytics Loops for AI Action Agents](https://iceberglakehouse.com/posts/active-analytics-loop-goal-directed-action-agents/) (2026-07-13): An in-depth exploration of active analytics loops for ai action agents - [The Chase-Catch Gap in Enterprise AI Agents](https://iceberglakehouse.com/posts/chase-catch-gap-agentic-ai-enterprise-scale-2026/) (2026-07-13): An in-depth exploration of the chase-catch gap in enterprise ai agents - [Semantic View Autopilot for AI Governance](https://iceberglakehouse.com/posts/semantic-view-autopilot-dynamic-metric-definition-governance/) (2026-07-13): An in-depth exploration of semantic view autopilot for ai governance - [The State of Agentic AI Standards in 2026: MCP, A2A, WebMCP, OSI, and the Protocol Stack Taking Shape](https://iceberglakehouse.com/posts/state-of-agentic-ai-standards-2026/) (2026-07-06): The agentic AI protocol stack is solidifying in 2026, MCP for tools, A2A for agents, WebMCP for the web, OSI for semantics, payments, identity. - [Building Closed-Loop Decision Agents: Moving from Passive BI Dashboards to Active Goal-Directed Workflows](https://iceberglakehouse.com/posts/closed-loop-decision-agents-passive-bi-active-workflows/) (2026-07-06): Dashboards are excellent at showing people what happened. They are less good at deciding what should happen next. That gap is where closed-loop de... - [Designing Your Own AI Harness: A Deep Dive Into the Architecture of Agent Loops, Tools, Context, and Control](https://iceberglakehouse.com/posts/designing-your-own-ai-harness/) (2026-07-06): A deep dive into custom AI harness architecture: model layers, tool design, context management, permissions, control budgets, persistence, orchestration. - [Enforcing Fine-Grained Security at Machine Speed: Dynamic Access Control for High-Frequency AI Agents](https://iceberglakehouse.com/posts/fine-grained-security-machine-speed-ai-agents/) (2026-07-06): AI agents change the security model for analytics. A human user may run a handful of queries, pause, interpret the answer, and ask a follow-up. An... - [The 2026-07-28 Model Context Protocol Release Candidate: What the Stateless Spec Means for Data Platforms](https://iceberglakehouse.com/posts/mcp-2026-07-28-stateless-spec-data-platforms/) (2026-07-06): The date in this topic matters. Today is July 6, 2026. A release candidate dated July 28, 2026 is still in the future. That means this article shou... - [Personal Context vs. Shared Context: A Deep Dive Into How Humans and Organizations Should Feed Their AI Agents](https://iceberglakehouse.com/posts/personal-vs-shared-context/) (2026-07-06): The most important discovery of the agent era fits in one sentence: most AI failures are context failures, not model failures. When your assistant... - [When Gatekeepers Panic: The Encyclopédie, Open AI Models, and the Politics of Accessible Knowledge](https://iceberglakehouse.com/posts/when-gatekeepers-panic/) (2026-07-06): The fight over open AI models mirrors the 18th-century suppression of Diderot's Encyclopédie, revealing the same pattern of institutional fear. - [Hermes Agent with DeepSeek V4 and Slack: Current Setup Guide](https://iceberglakehouse.com/posts/2026-05-hermes-agent-free-deepseek-setup/) (2026-05-25): Install Hermes Agent, choose a current DeepSeek provider, and connect Slack through Socket Mode with the tokens, scopes, events, and allowlist Hermes requires. - [Context Management Strategies for Zed: A Complete Guide to the High-Performance AI Code Editor](https://iceberglakehouse.com/posts/2026-03-context-zed/) (2026-03-07): Zed is a high-performance code editor built in Rust that prioritizes speed, simplicity, and real-time collaboration. Its AI integration is designed to be fas... - [Context Management Strategies for Windsurf: A Complete Guide to the AI Flow IDE](https://iceberglakehouse.com/posts/2026-03-context-windsurf/) (2026-03-07): Windsurf is an AI-powered IDE built on the VS Code foundation that introduces the concept of "Flows," a paradigm where the AI maintains deep awareness of you... - [Context Management Strategies for Perplexity AI: A Complete Guide to Research-First AI Conversations](https://iceberglakehouse.com/posts/2026-03-context-perplexity/) (2026-03-07): Perplexity AI occupies a unique position in the AI landscape: it is a research-first tool that combines conversational AI with real-time web search to produc... - [Context Management Strategies for OpenWork: A Complete Guide to the Desktop AI Agent Framework](https://iceberglakehouse.com/posts/2026-03-context-openwork/) (2026-03-07): OpenWork is a desktop-native AI agent framework designed for local, multi-step task execution on your computer. Unlike browser-based AI tools or terminal age... - [Context Management Strategies for Google Antigravity: A Complete Guide to the Agent-First IDE](https://iceberglakehouse.com/posts/2026-03-context-google-antigravity/) (2026-03-07): Google Antigravity is an agent-first IDE built by Google DeepMind's Advanced Agentic Coding team. It approaches context management differently from other AI ... - [Context Management Strategies for Gemini CLI: A Complete Guide to Terminal-Native AI Development](https://iceberglakehouse.com/posts/2026-03-context-gemini-cli/) (2026-03-07): Gemini CLI is an open-source terminal agent powered by Gemini models that operates directly in your command line. It brings Google's AI capabilities into the... - [Context Management Strategies for Gemini Web and NotebookLM: A Complete Guide to Google's AI Knowledge Ecosystem](https://iceberglakehouse.com/posts/2026-03-context-gemini-web-notebooklm/) (2026-03-07): Google's AI ecosystem for knowledge work consists of two deeply integrated tools: Gemini (the conversational AI at gemini.google.com) and NotebookLM (the res... - [Context Management Strategies for Claude Code: A Complete Guide for Developers](https://iceberglakehouse.com/posts/2026-03-context-claude-code/) (2026-03-07): Claude Code is a terminal-native agentic coding assistant that lives in your command line and operates directly on your codebase. Unlike chat-based interface... - [Context Management Strategies for Claude CoWork: A Complete Guide for Knowledge Workers](https://iceberglakehouse.com/posts/2026-03-context-claude-cowork/) (2026-03-07): Claude CoWork represents a fundamentally different approach to AI context management. Unlike chat interfaces where you send messages and receive responses, C... - [Context Management Strategies for Claude Desktop: A Complete Guide to MCP, Computer Use, and Local File Access](https://iceberglakehouse.com/posts/2026-03-context-claude-desktop/) (2026-03-07): Claude Desktop takes everything available in Claude Web and adds three capabilities that fundamentally change how you manage context: MCP server connections ... - [Context Management Strategies for Claude Web: A Complete Guide to Projects, Artifacts, and Intelligent Context](https://iceberglakehouse.com/posts/2026-03-context-claude-web/) (2026-03-07): Claude's web interface at claude.ai combines one of the largest context windows in the industry with a structured Project system that makes it genuinely usef... - [Codex Context Management: Context Windows, AGENTS.md, and Skills](https://iceberglakehouse.com/posts/2026-03-context-openai-codex/) (2026-03-07): Manage Codex context with layered AGENTS.md guidance, reusable skills, project configuration, focused prompts, and external tools when they are actually needed. - [Context Management Strategies for ChatGPT: A Complete Guide to Getting Better Results](https://iceberglakehouse.com/posts/2026-03-context-chatgpt/) (2026-03-07): Getting consistently useful results from ChatGPT requires more than writing good prompts. The real differentiator is how you manage context: the background i... - [Context Management Strategies for T3 Chat: A Complete Guide to the Unified Multi-Model AI Interface](https://iceberglakehouse.com/posts/2026-03-context-t3-chat/) (2026-03-07): T3 Chat is a modern web-based AI chat interface that gives you access to multiple AI models through a single unified platform. Its primary value proposition ... - [Context Management Strategies for VS Code with LLM Plugins: A Complete Guide to Building Your Own AI-Powered IDE](https://iceberglakehouse.com/posts/2026-03-context-vscode-llm-plugins/) (2026-03-07): Visual Studio Code is the most widely used code editor in the world, and its extensibility means you can integrate AI capabilities through a growing ecosyste... - [The Model Context Protocol (MCP) Explained: A Complete Guide to How Every Major AI Tool Connects to External Data](https://iceberglakehouse.com/posts/2026-03-context-mcp-deep-dive/) (2026-03-07): The Model Context Protocol (MCP) has become the universal standard for connecting AI models to external tools, data sources, and services. Originally open-so... - [What Are Recursive Language Models?](https://iceberglakehouse.com/posts/2026-01-recursive-langauge-models/) (2026-01-10): Recursive Language Models (RLMs) are a new class of language models that can call themselves to break down complex tasks into manageable parts. - [Building a Basic MCP Server with Python](https://iceberglakehouse.com/posts/2025-04-basics-of-making-mcp-server/) (2025-04-04): The Basics of Building a Basic MCP Server - [Crash Course on Developing AI Applications with LangChain](https://iceberglakehouse.com/posts/2025-02-crash-course-on-langchain/) (2025-02-01): A guide on building AI applications with LangChain, a framework for developing AI applications powered by Large Language Models (LLMs). ## Articles: Dremio - [How to Use Dremio with Zed: Connect, Query, and Build Data Apps](https://iceberglakehouse.com/posts/2026-03-aitool-zed/) (2026-03-05): Zed is an open-source, GPU-accelerated code editor written in Rust. It is designed for speed and collaboration, with a built-in AI assistant that supports mu... - [How to Use Dremio with Windsurf: Connect, Query, and Build Data Apps](https://iceberglakehouse.com/posts/2026-03-aitool-windsurf/) (2026-03-05): Windsurf is an AI-native code editor built as a fork of VS Code. Its standout feature is Cascade, an agentic AI system that plans and executes multi-step cod... - [How to Use Dremio with OpenWork: Connect, Query, and Build Data Apps](https://iceberglakehouse.com/posts/2026-03-aitool-openwork/) (2026-03-05): OpenWork is an open-source desktop AI agent built on the OpenCode engine. It runs entirely on your machine with your own API keys, giving you full control ov... - [How to Use Dremio with OpenCode: Connect, Query, and Build Data Apps](https://iceberglakehouse.com/posts/2026-03-aitool-opencode/) (2026-03-05): OpenCode is an open-source, terminal-based AI coding agent released under the MIT license. It provides a TUI with split panes, uses the Language Server Proto... - [How to Use Dremio with OpenAI Codex CLI: Connect, Query, and Build Data Apps](https://iceberglakehouse.com/posts/2026-03-aitool-openai-codex/) (2026-03-05): OpenAI Codex CLI is a terminal-based coding agent built in Rust. It reads your codebase, writes files, executes commands, and supports MCP for connecting to ... - [How to Use Dremio with JetBrains AI Assistant: Connect, Query, and Build Data Apps](https://iceberglakehouse.com/posts/2026-03-aitool-jetbrains-ai/) (2026-03-05): JetBrains AI Assistant is built into IntelliJ IDEA, PyCharm, DataGrip, and every JetBrains IDE. It provides AI chat, inline code generation, multi-file refac... - [How to Use Dremio with Google Antigravity: Connect, Query, and Build Data Apps](https://iceberglakehouse.com/posts/2026-03-aitool-google-antigravity/) (2026-03-05): Google Antigravity is an agent-first IDE built by Google DeepMind. Its autonomous agents plan multi-step tasks, write code, browse documentation, and iterate... - [How to Use Dremio with GitHub Copilot: Connect, Query, and Build Data Apps](https://iceberglakehouse.com/posts/2026-03-aitool-github-copilot/) (2026-03-05): GitHub Copilot is the most widely adopted AI coding assistant, integrated into VS Code, JetBrains IDEs, and the GitHub platform. Its agent mode allows Copilo... - [How to Use Dremio with Gemini CLI: Connect, Query, and Build Data Apps](https://iceberglakehouse.com/posts/2026-03-aitool-gemini-cli/) (2026-03-05): Gemini CLI is Google's open-source terminal-based AI agent. It runs directly in your terminal, powered by Gemini models with a 1-million token context window... - [How to Use Dremio with Cursor: Connect, Query, and Build Data Apps](https://iceberglakehouse.com/posts/2026-03-aitool-cursor/) (2026-03-05): Cursor is an AI-native code editor built as a fork of VS Code. It integrates AI directly into the editing experience with features like Chat, Composer (multi... - [How to Use Dremio with Claude CoWork: Connect, Query, and Build Data Apps](https://iceberglakehouse.com/posts/2026-03-aitool-claude-cowork/) (2026-03-05): Claude CoWork is Anthropic's desktop agentic assistant. Unlike Claude Code (a terminal coding agent), CoWork operates as a general-purpose autonomous agent t... - [How to Use Dremio with Claude Code: Connect, Query, and Build Data Apps](https://iceberglakehouse.com/posts/2026-03-aitool-claude-code/) (2026-03-05): Claude Code is Anthropic's terminal-based coding agent. It reads your files, writes code, runs commands, and maintains context across a session. Dremio is a ... - [How to Use Dremio with Amazon Kiro: Connect, Query, and Build Data Apps](https://iceberglakehouse.com/posts/2026-03-aitool-amazon-kiro/) (2026-03-05): Amazon Kiro is an agentic AI IDE from AWS that introduces spec-driven development to the coding workflow. Instead of jumping straight to code, Kiro helps you... - [Connect Apache Druid to Dremio Cloud: Add SQL Joins, AI, and Governance to Your Real-Time Analytics](https://iceberglakehouse.com/posts/2026-03-connector-apache-druid/) (2026-03-01): Apache Druid is a real-time analytics database designed for sub-second queries on high-ingestion-rate event data. Clickstream analytics, application monitori... - [Connect MongoDB to Dremio Cloud: SQL Analytics on Document Data](https://iceberglakehouse.com/posts/2026-03-connector-mongodb/) (2026-03-01): MongoDB is the most popular NoSQL document database. It stores data in flexible JSON-like documents, making it ideal for applications with evolving. - [Connect Vertica to Dremio Cloud: Federation for Analytics-Optimized Data](https://iceberglakehouse.com/posts/2026-03-connector-vertica/) (2026-03-01): Vertica is a columnar analytics database engineered for fast aggregate queries on large datasets. It was built from the ground up for analytical workloads : ... - [Connect Azure Synapse Analytics to Dremio Cloud: Multi-Cloud Data Warehouse Federation](https://iceberglakehouse.com/posts/2026-03-connector-azure-synapse/) (2026-03-01): Microsoft Azure Synapse Analytics combines big data analytics and enterprise data warehousing into a single Azure-integrated platform. If your organization h... - [Connect Snowflake to Dremio Cloud: Federate, Govern, and Accelerate Beyond Snowflake](https://iceberglakehouse.com/posts/2026-03-connector-snowflake/) (2026-03-01): Snowflake is a popular cloud data warehouse known for its separation of storage and compute, near-zero maintenance, and broad ecosystem. Many organizations h... - [Connect Google BigQuery to Dremio Cloud: Cross-Cloud Analytics Without Data Movement](https://iceberglakehouse.com/posts/2026-03-connector-google-bigquery/) (2026-03-01): Google BigQuery is Google Cloud's serverless data warehouse. If your organization uses Google Cloud Platform, BigQuery is where your analytics data, marketin... - [Connect Amazon Redshift to Dremio Cloud: Extend Your Warehouse with Federation and AI Analytics](https://iceberglakehouse.com/posts/2026-03-connector-amazon-redshift/) (2026-03-01): Amazon Redshift is AWS's managed data warehouse, designed for petabyte-scale analytics. If your organization chose Redshift for analytical workloads, you've ... - [Connect Azure Storage to Dremio Cloud: Query Your Microsoft Data Lake with SQL and AI](https://iceberglakehouse.com/posts/2026-03-connector-azure-storage/) (2026-03-01): Azure Storage is Microsoft's cloud storage platform, spanning Blob Storage, Azure Data Lake Storage Gen2 (ADLS Gen2), and Azure Files. If your organization u... - [Connect Amazon S3 to Dremio Cloud: Query Your Data Lake with SQL, Federation, and AI](https://iceberglakehouse.com/posts/2026-03-connector-amazon-s3/) (2026-03-01): Amazon S3 is the default landing zone for data in the cloud. Log files, Parquet datasets, CSV exports, JSON events, IoT telemetry, and raw data dumps : it al... - [Connect SAP HANA to Dremio Cloud: Unlock Analytics Beyond the SAP Ecosystem](https://iceberglakehouse.com/posts/2026-03-connector-sap-hana/) (2026-03-01): SAP HANA is the in-memory database platform that powers SAP S/4HANA, SAP BW/4HANA, and custom enterprise applications across finance, manufacturing, logistic... - [Connect IBM Db2 to Dremio Cloud: Modernize Mainframe Analytics with Federation and AI](https://iceberglakehouse.com/posts/2026-03-connector-ibm-db2/) (2026-03-01): IBM Db2 is the relational database that powers critical applications across banking, insurance, government, healthcare, and manufacturing. For organizations ... - [Connect Microsoft SQL Server to Dremio Cloud: Federate Enterprise Data Without ETL](https://iceberglakehouse.com/posts/2026-03-connector-microsoft-sql-server/) (2026-03-01): Microsoft SQL Server is one of the most widely deployed enterprise databases in the world. ERP systems, CRM platforms, financial applications, and custom bus... - [Extract Structured Data from Text with Dremio's AI_GENERATE Function](https://iceberglakehouse.com/posts/2026-03-ai-ai-generate/) (2026-03-01): Unstructured text is the most underused data in most organizations. Customer emails sit in inboxes. Contract notes live in text fields. Meeting summaries exi... - [Connect Oracle Database to Dremio Cloud: Enterprise Analytics Without Data Movement](https://iceberglakehouse.com/posts/2026-03-connector-oracle/) (2026-03-01): Oracle Database runs the most critical enterprise applications in the world : ERP systems, financial ledgers, supply chain management, and HR platforms. Thes... - [Generate Summaries and Insights with Dremio's AI_COMPLETE Function](https://iceberglakehouse.com/posts/2026-03-ai-ai-complete/) (2026-03-01): Every data team has a version of this problem: a table full of raw data that needs human-readable summaries, translations, or narrative descriptions. Product... - [Connect MySQL to Dremio Cloud: Federated Analytics Without ETL](https://iceberglakehouse.com/posts/2026-03-connector-mysql/) (2026-03-01): MySQL runs more web applications, SaaS platforms, and e-commerce backends than any other database. It's fast for transactional reads and writes, but it becom... - [Classify Your Data with SQL: A Hands-On Guide to Dremio's AI_CLASSIFY Function](https://iceberglakehouse.com/posts/2026-03-ai-ai-classify/) (2026-03-01): Most classification workflows require exporting data to Python, running a model, and importing results back into your warehouse. Dremio's `AI_CLASSIFY` funct... - [Connect PostgreSQL to Dremio Cloud: Query, Federate, and Accelerate Your Data](https://iceberglakehouse.com/posts/2026-03-connector-postgresql/) (2026-03-01): PostgreSQL powers more production applications than almost any other open-source database. It's where your customer records, transaction logs, product catalo... - [Connect AWS Glue Data Catalog to Dremio Cloud: Query and Manage Your AWS Iceberg Tables](https://iceberglakehouse.com/posts/2026-03-connector-aws-glue/) (2026-03-01): AWS Glue Data Catalog is AWS's managed metadata service for data lakes. It stores table definitions, schemas, partition information, and statistics for data ... - [Connect Snowflake Open Catalog to Dremio Cloud: Multi-Engine Iceberg Analytics](https://iceberglakehouse.com/posts/2026-03-connector-snowflake-open-catalog/) (2026-03-01): Snowflake Open Catalog is Snowflake's managed implementation of the Apache Iceberg REST catalog specification, based on the open-source Apache Polaris projec... - [Connect Databricks Unity Catalog to Dremio Cloud: Query Delta Lake Tables with Federation and AI](https://iceberglakehouse.com/posts/2026-03-connector-unity-catalog/) (2026-03-01): Databricks Unity Catalog is Databricks' governance layer for data and AI assets. It manages Delta Lake tables, machine learning models, feature stores, and o... - [Connect Any Iceberg REST Catalog to Dremio Cloud: Universal Lakehouse Access](https://iceberglakehouse.com/posts/2026-03-connector-iceberg-rest-catalog/) (2026-03-01): The Apache Iceberg REST Catalog specification defines a standard HTTP API for managing Iceberg table metadata. Any catalog implementation that conforms to th... - [Dremio's Built-in Open Catalog: Your Zero-Configuration Apache Iceberg Lakehouse](https://iceberglakehouse.com/posts/2026-03-connector-dremio-open-catalog/) (2026-03-01): Every Dremio Cloud account starts with a built-in Open Catalog : a fully managed Apache Iceberg catalog with integrated storage. When you create a Dremio Clo... - [Connect Dremio Software to Dremio Cloud: Hybrid Federation Across Deployments](https://iceberglakehouse.com/posts/2026-03-connector-dremio-to-dremio/) (2026-03-01): Dremio Cloud can connect to Dremio Software (self-managed) instances as a federated data source. This creates a hybrid deployment where Dremio Cloud serves a... ## Articles: Data Engineering - [Running an Iceberg Lakehouse on Kubernetes](https://iceberglakehouse.com/posts/iceberg-on-kubernetes/) (2026-09-10): Catalog, maintenance, and compaction as Kubernetes workloads: scheduling classes, job structure, credential flow, and the failures that come from the interaction. - [What to Assert When You Test an Iceberg Pipeline](https://iceberglakehouse.com/posts/iceberg-pipeline-assertions/) (2026-09-10): Fixtures, in-memory catalogs, and golden metadata: the assertions that catch wrong rows, unsafe reruns, schema drift, and concurrent-write corruption in CI. - [Kafka Connect to Iceberg: How the Commit Actually Works](https://iceberglakehouse.com/posts/kafka-connect-iceberg-commits/) (2026-09-10): Exactly-once semantics in the Iceberg sink connector: the coordinator, the control topic, offsets stored inside Iceberg snapshots, and where duplicates still get in. - [What a Query Costs](https://iceberglakehouse.com/posts/lakehouse-unit-economics/) (2026-09-10): Four meters, their real proportions, and how to attribute compute to a query, a table, and a team so a platform can answer what a dashboard costs to run. - [Data Quality Tooling Compared: Great Expectations, Soda, dbt Tests, and Anomaly Detection](https://iceberglakehouse.com/posts/data-quality-tooling-compared/) (2026-09-02): A comparison of Great Expectations, Soda, dbt tests, and anomaly detection, and a layered design that uses each where it fits. - [dbt on Iceberg: Incremental Models on Open Tables](https://iceberglakehouse.com/posts/dbt-on-iceberg-incremental-models/) (2026-09-02): How dbt incremental materializations map to Iceberg operations, and the configuration, predicates, and maintenance that keep them healthy. - [The Lakehouse Ingestion Tool Landscape: Fivetran, Airbyte, dlt, and CDC vs Batch](https://iceberglakehouse.com/posts/lakehouse-ingestion-tools/) (2026-09-02): How Fivetran, Airbyte, dlt, and CDC and streaming tools land well-behaved Apache Iceberg tables, and how to choose and maintain them. - [Logs, Traces, and Metrics as Tables: Building an OpenTelemetry Data Lake on Iceberg](https://iceberglakehouse.com/posts/opentelemetry-data-lake-on-iceberg/) (2026-09-02): Building an OpenTelemetry data lake on Iceberg: schemas for spans, logs, and metrics, ingestion, query patterns, and retention. - [Orchestration in 2026: Airflow 3 vs Dagster vs Prefect vs Event-Driven](https://iceberglakehouse.com/posts/orchestration-in-2026/) (2026-09-02): Where Airflow 3, Dagster, Prefect, and event-driven triggering stand for lakehouse pipelines in 2026, after the Prefect acquisition of Dagster. - [Schema Registries and Event Schemas: Avro, Protobuf, and JSON Schema on the Way Into the Lakehouse](https://iceberglakehouse.com/posts/schema-registries-and-event-schemas/) (2026-09-02): How Avro, Protobuf, and JSON Schema evolve through a registry, and how that maps to the schema evolution rules of Iceberg. - [Synthetic Data in the Lakehouse: Generation, Governance, and Testing](https://iceberglakehouse.com/posts/synthetic-data-in-the-lakehouse/) (2026-09-02): What synthetic data in a lakehouse is for, the generation methods, how to preserve fidelity, and where synthetic tables belong. - [Arrow Flight SQL and ADBC: Why the Database Driver Is the Slowest Part of Your Query](https://iceberglakehouse.com/posts/arrow-flight-sql-adbc-connectivity/) (2026-08-25): JDBC and ODBC often dominate large-result time. Flight SQL and ADBC keep data columnar from server to client, with Python, Go, and Rust examples. - [Query Routing at Machine Scale: Dynamic Workload Distribution Across Lakehouse Engines](https://iceberglakehouse.com/posts/query-routing-lakehouse-engines/) (2026-08-25): Route each lakehouse query by shape, not by sender. Signals, rules, and how to keep dashboards, batch jobs, and agents from sharing one engine. - [Semantic Layer Federation: One Meaning for Data That Lives Everywhere](https://iceberglakehouse.com/posts/semantic-layer-federation-multi-cloud/) (2026-08-19): Build a federated semantic layer across multi-cloud data so one set of governed metric definitions serves BI tools, dashboards, and AI agents identically. - [Apache Arrow Flight and ADBC, and Why Database Connectivity Finally Went Columnar](https://iceberglakehouse.com/posts/arrow-flight-adbc-explained/) (2026-08-06): Arrow Flight and ADBC move database results as columnar data, ending the row-oriented bottleneck between engines and applications. Here's how. - [Why AI Agents Fail on Raw Data, and What to Give Them Instead](https://iceberglakehouse.com/posts/data-products-for-ai-agents/) (2026-07-28): Agents fail on raw lake data because business rules live in people's heads. Data products with semantic contracts fix this at the source. - [Freshness Is a Contract, Not a Note on a Dashboard](https://iceberglakehouse.com/posts/freshness-as-a-contract/) (2026-07-25): Data freshness needs to become an engineering contract with a measurable value, an owner, and consequences. - [The Whole Lakehouse Fits on Your Laptop Now](https://iceberglakehouse.com/posts/laptop-scale-lakehouse/) (2026-07-25): Consumer hardware, columnar formats, single-node engines, and the Iceberg REST catalog crossed a threshold: a large share of cluster work now runs locally. - [Iceberg's Next Version Depends on Decisions Being Made in Parquet](https://iceberglakehouse.com/posts/parquet-constrains-iceberg/) (2026-07-25): The Iceberg and Parquet specifications are co-evolving. Several of the most consequential Iceberg v4 proposals are waiting on Parquet work. - [Building a Lakehouse That Stays Inside the Border](https://iceberglakehouse.com/posts/sovereign-lakehouse/) (2026-07-25): Residency is a storage location. Sovereignty is who can compel access, who operates the systems, and whether you can leave. - [Table Maintenance Stopped Being a Product](https://iceberglakehouse.com/posts/table-maintenance-economics/) (2026-07-25): Iceberg table maintenance commoditized when every platform started shipping it. What the six operations are, what they cost, and the observability you. - [Three Vendors Are Rebuilding the Path From Transaction to Agent](https://iceberglakehouse.com/posts/transaction-to-agent-race/) (2026-07-25): Databricks, Snowflake, and SAP are closing the gap between operational databases and analytical platforms through acquisition, betting on different layers. - [What Zero-Copy Actually Costs](https://iceberglakehouse.com/posts/zero-copy-actually-costs/) (2026-07-25): Six architectures share the phrase zero-copy, and they have different costs, failure modes, and governance stories. - [The Breakdown of the Open Lakehouse in 2026: Iceberg, Arrow, Polaris, Parquet, and Ossie, and How to Actually Build One](https://iceberglakehouse.com/posts/open-lakehouse-breakdown-2026/) (2026-07-24): A detailed breakdown of the five Apache projects powering the open lakehouse in 2026, Parquet, Iceberg, Polaris, Arrow, and Ossie, and how to build one. - [Understanding the RAMpocalypse: An Economic Autopsy of the Great Memory Shortage and What Comes Next](https://iceberglakehouse.com/posts/rampocalypse-economics/) (2026-07-24): An economic autopsy of the 2026 memory shortage, why DRAM prices surged 90% in a quarter, and what supply, demand, and market structure mean. - [Block vs. Object Storage: A Deep Dive Into the Foundation of Modern Data, and How the Lakehouse Made the Slow Option Fast](https://iceberglakehouse.com/posts/block-vs-object-storage-deep-dive/) (2026-07-06): Here is one of the strangest and most consequential plot twists in the history of data infrastructure: over the past decade, the analytics industry... - [Deterministic Data Engineering With AI Harnesses: Using Claude Code, Codex, Antigravity, and OpenCode for Data Work You Can Actually Trust](https://iceberglakehouse.com/posts/deterministic-data-engineering-with-ai-harnesses/) (2026-07-06): How to use AI agent harnesses for data engineering without losing determinism, reproducibility, and trust in your data pipelines and analytics. - [Preparing Your Data Lakehouse for the EU AI Act: Auditable Lineage and Data Provenance](https://iceberglakehouse.com/posts/eu-ai-act-data-lakehouse-lineage-provenance-compliance/) (2026-07-06): The EU AI Act changes the conversation around AI architecture because it makes trust operational. It is not enough to say that an AI system is usef... - [A Deep Dive Into File Compression: How Data Gets Smaller, Why Codecs Differ, and What to Actually Use in the Lakehouse](https://iceberglakehouse.com/posts/file-compression-deep-dive/) (2026-07-06): Somewhere in your data platform right now, a single configuration property is quietly deciding a meaningful percentage of your storage bill, your q... - [The File Format Renaissance: Parquet, Lance, Vortex, Nimble, BtrBlocks, and the New Physics of Columnar Storage](https://iceberglakehouse.com/posts/file-formats-renaissance/) (2026-07-06): For a decade, the file format layer was the most settled real estate in data. Apache Parquet held the analytical world, ORC held the Hive legacy es... - [File Encryption for the Lakehouse: The Terminology, the Machinery, and the Hard Problem of Interoperable Encrypted Tables](https://iceberglakehouse.com/posts/lakehouse-encryption-deep-dive/) (2026-07-06): For years, the open lakehouse had an honest gap that practitioners whispered about and slide decks skipped: encryption. Not the checkbox kind, ever... - [Open Source Foundations, Explained: What Apache, Linux, Eclipse, and Their Peers Actually Do, and Why Governance Differences Matter](https://iceberglakehouse.com/posts/open-source-foundations-explained/) (2026-07-06): Writing about open data and AI means repeating the same phrases over and over: donated to the Apache Software Foundation, incubating at the Linux F... - [Operational vs. Analytical Systems: Why the Oldest Divide in Data Exists, What Physics Enforces It, and the Honest Truth About Hybrid Systems](https://iceberglakehouse.com/posts/operational-vs-analytical-systems/) (2026-07-06): Every data architecture ever drawn contains the same fault line, so old and so universal that most engineers stop seeing it: on one side, the syste... - [Migrating Proprietary Warehouses to Open Lakehouses: The 2026 Playbook for Zero-Copy Metadata Translation](https://iceberglakehouse.com/posts/proprietary-warehouse-open-lakehouse-zero-copy-metadata-translation/) (2026-07-06): Every warehouse migration sounds simpler before the first inventory. Then the team discovers old dashboards, hidden dependencies, undocumented stor... - [Data Platform Native AI Agent Tooling in 2026](https://iceberglakehouse.com/posts/data-platform-ai-agent-tooling/) (2026-05-31): A comprehensive comparison of AI agent tooling across Dremio, Snowflake, Databricks, Microsoft Fabric, AWS, Google Cloud, ClickHouse, VeloDB, SpiceAI. - [Automating Table Maintenance Before Small Files Accumulate](https://iceberglakehouse.com/posts/2026-05-24-automating-table-maintenance/) (2026-05-24): Learn how Databricks Predictive Optimization, AWS S3 Tables, and Iceberg native actions automate compaction and snapshot management before small files. - [Choosing the Right Iceberg Control Plane: Polaris vs. Unity Catalog vs. Cloud REST](https://iceberglakehouse.com/posts/2026-05-24-choosing-iceberg-control-plane/) (2026-05-24): Choosing an Apache Iceberg catalog? Compare open-source Apache Polaris, open Unity Catalog, and managed cloud REST control planes to unify your lakehouse. - [Clean Rooms for Privacy-Preserving Analytics](https://iceberglakehouse.com/posts/2026-05-24-clean-rooms-privacy/) (2026-05-24): Data clean rooms enable secure multi-party analytics without sharing raw data. Learn how Databricks Clean Rooms, AWS Clean Rooms, and BigQuery. - [Building Composable Query Engines with Rust Runtimes](https://iceberglakehouse.com/posts/2026-05-24-composable-query-engines/) (2026-05-24): Apache DataFusion, Velox, and Substrait form the foundation of modern composable query engine stacks. - [Data Mesh After the Hype: What Actually Works](https://iceberglakehouse.com/posts/2026-05-24-data-mesh-after-hype/) (2026-05-24): Three years after Zhamak Dehghani's original papers, data mesh has proven valuable in specific organizational contexts and impractical in others. - [How dbt Fusion Reshapes Analytics Engineering](https://iceberglakehouse.com/posts/2026-05-24-dbt-fusion-analytics-engineering/) (2026-05-24): dbt Fusion entered public beta in May 2025 with a Rust-powered runtime that changes how analytics engineers develop, validate, and deploy SQL models. - [Using DuckDB and Polars to Query Iceberg Tables](https://iceberglakehouse.com/posts/2026-05-24-duckdb-polars-iceberg/) (2026-05-24): DuckDB 1.4 LTS and Polars streaming engine now both support reading and writing Apache Iceberg tables. Learn how to use them for lakehouse analytics in 2025. - [FinOps for Data Warehouses with Open Billing Data](https://iceberglakehouse.com/posts/2026-05-24-finops-warehouse-cost/) (2026-05-24): The FOCUS 1. 3 specification and native warehouse cost views make real-time cost attribution practical. - [Designing Governed RAG on Data Products](https://iceberglakehouse.com/posts/2026-05-24-governed-rag-data-products/) (2026-05-24): Enterprise RAG architecture that trusts its own data requires governance at the retrieval layer. - [What Iceberg V3 Advances Mean for CDC Pipelines](https://iceberglakehouse.com/posts/2026-05-24-iceberg-cdc-pipelines/) (2026-05-24): Apache Iceberg V3 brings deletion vectors and row lineage that reshape CDC pipeline design. Learn what these features mean for your streaming data architecture. - [Lance and Iceberg for Multimodal AI Data](https://iceberglakehouse.com/posts/2026-05-24-lance-iceberg-multimodal/) (2026-05-24): LanceDB and Apache Iceberg serve complementary roles in a multimodal AI lakehouse. Learn when to use Lance for embeddings and random access, and Iceberg. - [Bringing MLflow and Data Pipelines Closer Together](https://iceberglakehouse.com/posts/2026-05-24-mlflow-data-pipelines/) (2026-05-24): MLflow 3 extends observability from classic ML experiments to GenAI tracing and data pipeline lineage. - [OpenLineage as the Spine of Data Observability](https://iceberglakehouse.com/posts/2026-05-24-openlineage-observability/) (2026-05-24): OpenLineage provides a standard API for collecting pipeline lineage across Airflow, Spark, Flink, and dbt. - [When Paimon Beats Iceberg for Mutable Streams](https://iceberglakehouse.com/posts/2026-05-24-paimon-vs-iceberg-mutable-streams/) (2026-05-24): Apache Paimon uses LSM-Tree storage for native CDC upserts without restart. Learn when Paimon outperforms Iceberg for high-churn mutable streaming workloads. - [Policy as Code for Lakehouse Governance](https://iceberglakehouse.com/posts/2026-05-24-policy-as-code-governance/) (2026-05-24): OPA, ABAC, row filters, and column masks make lakehouse governance programmable and scalable. - [Why Semantic Layers Make Enterprise Text-to-SQL Safer](https://iceberglakehouse.com/posts/2026-05-24-semantic-layers-text-to-sql/) (2026-05-24): Text-to-SQL accuracy jumps from 40% to 85-95% when grounded in a semantic layer. Learn how Dremio, Snowflake Cortex Analyst, and dbt Semantic Layer. - [Choosing Vector Stores for Retrieval Workloads](https://iceberglakehouse.com/posts/2026-05-24-vector-stores-retrieval/) (2026-05-24): pgvector, Milvus, Weaviate, and LanceDB each make different tradeoffs on index type, hybrid search, scale, and operational complexity. - [Single-Node Data Engineering: DuckDB, DataFusion, Polars, and LakeSail](https://iceberglakehouse.com/posts/2026-05-23-single-node-data-engineering-duckdb-datafusion-polars-lakesail/) (2026-05-23): Optimize single-node data engineering with DuckDB, DataFusion, Polars, and LakeSail. Compare architectures and learn when to transition to Dremio MPP. - [Setting Up an AWS-Native Open Lakehouse: Querying Apache Iceberg with AWS Athena and AWS Glue Catalog](https://iceberglakehouse.com/posts/2026-05-22-apache-iceberg-aws-athena-glue/) (2026-05-22): A comprehensive guide to building an open, high-performance lakehouse on AWS using Apache Iceberg, AWS Glue Catalog, Amazon S3, and S3 Tables, with query. - [Maintaining Apache Iceberg Tables: Compaction, Snapshot Expiration, and Orphan File Cleanup](https://iceberglakehouse.com/posts/2026-05-22-apache-iceberg-maintenance-compaction/) (2026-05-22): An in-depth guide to orchestrating maintenance operations on Apache Iceberg tables, covering bin-packing, sort-based, Z-Order compaction, snapshot. - [Migrating to Apache Iceberg: Strategies for Every Source System](https://iceberglakehouse.com/posts/2026-04-29-iceberg-masterclass-15/) (2026-04-29): Migrate to Iceberg from Hive, data warehouses, or raw files using in-place migration, full rewrite, or the zero-downtime view swap pattern. - [Hands-On with Apache Iceberg Using Dremio Cloud](https://iceberglakehouse.com/posts/2026-04-29-iceberg-masterclass-14/) (2026-04-29): A practical walkthrough of creating, querying, and optimizing Iceberg tables on Dremio Cloud, from account setup to AI-powered analytics. - [Approaches to Streaming Data into Apache Iceberg Tables](https://iceberglakehouse.com/posts/2026-04-29-iceberg-masterclass-13/) (2026-04-29): Stream data into Iceberg with Spark Structured Streaming, Flink, or Kafka Connect. Here is how each works and the trade-offs between latency and maintenance. - [Using Apache Iceberg with Python and MPP Query Engines](https://iceberglakehouse.com/posts/2026-04-29-iceberg-masterclass-12/) (2026-04-29): Access Iceberg tables from Python with PyIceberg, DuckDB, and Polars, or through MPP engines like Dremio, Spark, and Trino. Here is how each approach works. - [Apache Iceberg Metadata Tables: Querying the Internals](https://iceberglakehouse.com/posts/2026-04-29-iceberg-masterclass-11/) (2026-04-29): Iceberg metadata tables let you query snapshots, files, manifests, and partitions using SQL. Here is every metadata table and how to use them. - [Maintaining Apache Iceberg Tables: Compaction, Expiry, and Cleanup](https://iceberglakehouse.com/posts/2026-04-29-iceberg-masterclass-10/) (2026-04-29): Keep Iceberg tables fast with compaction, snapshot expiry, orphan cleanup, and manifest rewriting. Here is when and how to run each operation. - [Concurrency, Isolation, and MVCC: How Engines Handle Contention](https://iceberglakehouse.com/posts/2026-04-29-query-engine-10/) (2026-04-29): Databases handle concurrent access using locks, MVCC, or optimistic concurrency control. Here is how each approach works and what tradeoffs each creates. - [How Data Lake Table Storage Degrades Over Time](https://iceberglakehouse.com/posts/2026-04-29-iceberg-masterclass-09/) (2026-04-29): Iceberg tables degrade through small files, orphan files, metadata bloat, sort order decay, and partition skew. Here is how to diagnose each problem. - [Hash, Sort-Merge, Broadcast: How Distributed Joins Work](https://iceberglakehouse.com/posts/2026-04-29-query-engine-09/) (2026-04-29): Distributed joins move data across the network using shuffle, broadcast, or co-location strategies. Here is how each works and when engines choose which. - [When Catalogs Are Embedded in Storage](https://iceberglakehouse.com/posts/2026-04-29-iceberg-masterclass-08/) (2026-04-29): S3 Tables and MinIO AI Stor embed the Iceberg catalog directly in the storage layer. Here is when embedded catalogs make sense and when they do not. - [Partitioning, Sharding, and Data Distribution Strategies](https://iceberglakehouse.com/posts/2026-04-29-query-engine-08/) (2026-04-29): Hash partitioning distributes data evenly. Range partitioning enables fast range scans. Both create tradeoffs. - [What Are Lakehouse Catalogs? The Role of Catalogs in Apache Iceberg](https://iceberglakehouse.com/posts/2026-04-29-iceberg-masterclass-07/) (2026-04-29): Lakehouse catalogs store metadata pointers, manage namespaces, and enforce access control. Here is the complete catalog landscape from Polaris to Glue. - [Buffer Pools, Caches, and the Memory Hierarchy](https://iceberglakehouse.com/posts/2026-04-29-query-engine-07/) (2026-04-29): Databases use buffer pools, column caches, and result caches to keep hot data in RAM. Here is how each caching strategy works and what happens when data. - [Writing to an Apache Iceberg Table: How Commits and ACID Actually Work](https://iceberglakehouse.com/posts/2026-04-29-iceberg-masterclass-06/) (2026-04-29): Here is exactly how an engine writes to an Iceberg table, step by step, from data files through the atomic commit that makes ACID guarantees possible. - [Volcano, Vectorized, Compiled: How Engines Execute Your Query](https://iceberglakehouse.com/posts/2026-04-29-query-engine-06/) (2026-04-29): The Volcano model processes one row at a time. Vectorized execution processes batches with SIMD. Code generation fuses operators into compiled code. - [Hidden Partitioning: How Iceberg Eliminates Accidental Full Table Scans](https://iceberglakehouse.com/posts/2026-04-29-iceberg-masterclass-05/) (2026-04-29): Iceberg's hidden partitioning separates physical layout from user queries using transform functions. - [Inside the Query Optimizer: How Engines Pick a Plan](https://iceberglakehouse.com/posts/2026-04-29-query-engine-05/) (2026-04-29): Query optimizers transform SQL into execution plans using rule-based rewrites, cost-based search, and adaptive runtime adjustments. - [Partition Evolution: Change Your Partitioning Without Rewriting Data](https://iceberglakehouse.com/posts/2026-04-29-iceberg-masterclass-04/) (2026-04-29): Iceberg lets you change partition schemes without rewriting data. Here is how partition evolution works internally and why Hive-style partitioning could. - [B-Trees, LSM Trees, and the Indexing Tradeoff Spectrum](https://iceberglakehouse.com/posts/2026-04-29-query-engine-04/) (2026-04-29): B-trees balance reads and writes for OLTP. LSM trees maximize write throughput. Bitmap indexes accelerate OLAP filtering. Here is when to use each. - [Performance and Apache Iceberg's Metadata](https://iceberglakehouse.com/posts/2026-04-29-iceberg-masterclass-03/) (2026-04-29): Iceberg's three-layer metadata tree eliminates directory listing and enables multi-level data skipping. Here is how scan planning actually works. - [How Databases Organize Data on Disk: Pages, Blocks, and File Formats](https://iceberglakehouse.com/posts/2026-04-29-query-engine-03/) (2026-04-29): Databases structure data on disk as heap files, sorted files, or LSM trees, then wrap it in formats like Parquet with metadata that lets engines skip. - [The Metadata Structure of Modern Table Formats](https://iceberglakehouse.com/posts/2026-04-29-iceberg-masterclass-02/) (2026-04-29): Iceberg uses a metadata tree, Delta Lake uses a transaction log, Hudi uses a timeline. Here is exactly how each format organizes metadata and why it matters. - [Row vs. Column: How Storage Layout Shapes Everything](https://iceberglakehouse.com/posts/2026-04-29-query-engine-02/) (2026-04-29): Row stores keep records together for fast transactions. Column stores keep field values together for fast analytics. - [What Are Table Formats and Why Were They Needed?](https://iceberglakehouse.com/posts/2026-04-29-iceberg-masterclass-01/) (2026-04-29): Table formats like Apache Iceberg solved the ACID, schema, and performance problems that turned data lakes into data swamps. Here is how each one works. - [How Query Engines Think: The Tradeoffs Behind Every Data System](https://iceberglakehouse.com/posts/2026-04-29-query-engine-01/) (2026-04-29): Every database is a collection of engineering tradeoffs. Learn the 9 design decisions that shape how query engines store, index, and process your data. - [Agentic Analytics on the Apache Lakehouse](https://iceberglakehouse.com/posts/2026-03-07-agentic-analytics/) (2026-03-07): If you grant a Large Language Model direct access to a raw Amazon S3 bucket filled with Parquet files, it will fail to answer your business questions.... - [What is Apache Arrow? Erasing the Serialization Tax](https://iceberglakehouse.com/posts/2026-03-07-apache-arrow/) (2026-03-07): If you pull a million records from a database into a Python notebook, the query runs instantly, but the transfer feels endlessly slow. Your compute en... - [What is Apache Iceberg? The Table Format Revolution](https://iceberglakehouse.com/posts/2026-03-07-apache-iceberg/) (2026-03-07): If you drop ten thousand Parquet files into an S3 bucket, you have a data swamp. You do not have a database. To run SQL queries against those files sa... - [What is Apache Parquet? Columns, Encoding, and Performance](https://iceberglakehouse.com/posts/2026-03-07-apache-parquet/) (2026-03-07): If you ask a data analyst to calculate the average transaction amount for the month of July using a massive CSV file, the compute engine must read eve... - [What is Apache Polaris? Unifying the Iceberg Ecosystem](https://iceberglakehouse.com/posts/2026-03-07-apache-polaris/) (2026-03-07): Treating thousands of Parquet files as a unified database table requires a brain. Apache Iceberg provides the metadata structure to do this, but. - [Apache Software Foundation: History, Purpose, and Process](https://iceberglakehouse.com/posts/2026-03-07-apache-software-foundation/) (2026-03-07): If you build a modern data lakehouse, you inevitably stack Apache Iceberg, Apache Parquet, and Apache Arrow. These projects dictate how you store, que... - [Assembling the Apache Lakehouse: The Modular Architecture](https://iceberglakehouse.com/posts/2026-03-07-assembling-apache-lakehouse/) (2026-03-07): For decades, the standard data architecture was monolithic. When you bought a data warehouse, you bought a single box where the vendor tightly coupled... - [Data Engineering Best Practices: The Complete Checklist](https://iceberglakehouse.com/posts/2026-02-debp-de-best-practices-checklist/) (2026-02-18): Best practices documents are easy to write and hard to use. They list principles without context, advice without prioritization, and rules without explaining... - [Data Modeling Best Practices: 7 Mistakes to Avoid](https://iceberglakehouse.com/posts/2026-02-dm-data-modeling-best-practices/) (2026-02-18): A bad data model doesn't announce itself. It hides behind slow dashboards, conflicting numbers, confused analysts, and AI agents that generate wrong SQL. By ... - [Semantic Layer Best Practices: 7 Mistakes to Avoid](https://iceberglakehouse.com/posts/2026-02-sl-semantic-layer-best-practices/) (2026-02-18): Semantic layers don't fail because the technology is wrong. They fail because of design decisions made in the first two weeks : choices that seem reasonable ... - [Pipeline Observability: Know When Things Break](https://iceberglakehouse.com/posts/2026-02-debp-observability-monitoring/) (2026-02-18): An analyst messages you on Slack - "The revenue numbers look wrong. Is the pipeline broken?" You check the orchestrator - all green. - [Data Vault Modeling: Hubs, Links, and Satellites](https://iceberglakehouse.com/posts/2026-02-dm-data-vault-modeling/) (2026-02-18): Dimensional modeling works well when your source systems are stable and your business questions are predictable. But what happens when sources change constan... - [How a Self-Documenting Semantic Layer Reduces Data Team Toil](https://iceberglakehouse.com/posts/2026-02-sl-self-documenting-semantic-layer/) (2026-02-18): Every data team knows documentation is important. And almost every data team has a backlog of undocumented tables, unlabeled columns, and outdated descriptio... - [Testing Data Pipelines: What to Validate and When](https://iceberglakehouse.com/posts/2026-02-debp-testing-data-pipelines/) (2026-02-18): Ask an application developer how they test their code and they'll describe unit tests, integration tests, CI/CD pipelines, and coverage metrics. Ask a data e... - [Denormalization: When and Why to Flatten Your Data](https://iceberglakehouse.com/posts/2026-02-dm-denormalization-when-why/) (2026-02-18): Normalization is the first rule taught in database design. Eliminate redundancy. Store each fact once. Use foreign keys. It's the right rule for transactiona... - [Headless BI: How a Universal Semantic Layer Replaces Tool-Specific Models](https://iceberglakehouse.com/posts/2026-02-sl-headless-bi-semantic-layer/) (2026-02-18): Your organization uses Tableau for executive dashboards, Power BI for operational reports, and Python notebooks for data science. Revenue is defined in Table... - [Partition and Organize Data for Performance](https://iceberglakehouse.com/posts/2026-02-debp-partition-and-organize/) (2026-02-18): A table with 500 million rows takes 45 seconds to query. After partitioning it by date, the same query : filtering on a single day, returns in 2 seconds. - [Data Modeling for Analytics: Optimize for Queries, Not Transactions](https://iceberglakehouse.com/posts/2026-02-dm-data-modeling-for-analytics/) (2026-02-18): The data model that runs your production application is almost never the right model for analytics. Transactional systems are designed for fast writes : inse... - [Data Virtualization and the Semantic Layer: Query Without Copying](https://iceberglakehouse.com/posts/2026-02-sl-data-virtualization-semantic-layer/) (2026-02-18): Every data pipeline you build to move data from one system to another costs you three things: time to build it, money to run it, and freshness you lose while... - [Batch vs. Streaming: Choose the Right Processing Model](https://iceberglakehouse.com/posts/2026-02-debp-batch-vs-streaming/) (2026-02-18): "We need real-time data. " This is one of the most expensive sentences in data engineering - because it''s rarely true, and implementing it when it''s not. - [Slowly Changing Dimensions: Types 1-3 with Examples](https://iceberglakehouse.com/posts/2026-02-dm-slowly-changing-dimensions/) (2026-02-18): Dimensions change. A customer moves cities. A product gets reclassified. An employee changes departments. How your data model handles these changes determine... - [The Role of the Semantic Layer in Data Governance](https://iceberglakehouse.com/posts/2026-02-sl-semantic-layer-data-governance/) (2026-02-18): Most organizations have a data governance policy. It lives in a Confluence page. It defines who owns what data, what terms mean, and who should have access. ... - [Schema Evolution Without Breaking Consumers](https://iceberglakehouse.com/posts/2026-02-debp-schema-evolution/) (2026-02-18): A source team renames a column from `user_id` to `customer_id`. Twelve hours later, five dashboards show blank values, two ML pipelines fail, and the data en... - [Dimensional Modeling: Facts, Dimensions, and Grains](https://iceberglakehouse.com/posts/2026-02-dm-dimensional-modeling/) (2026-02-18): Dimensional modeling is the most widely used approach for organizing analytics data. Developed by Ralph Kimball, it structures data into two types of tables:... - [Why Your AI Initiatives Fail Without a Semantic Layer](https://iceberglakehouse.com/posts/2026-02-sl-why-ai-fails-without-semantic-layer/) (2026-02-18): Your team builds an AI agent. It connects to your data warehouse. A product manager types "What was revenue last quarter?" and gets a number. The number is w... - [Idempotent Pipelines: Build Once, Run Safely Forever](https://iceberglakehouse.com/posts/2026-02-debp-idempotent-pipelines/) (2026-02-18): A pipeline runs, processes 100,000 records, and loads them into the target table. Then it fails on a downstream step. The orchestrator retries the entire job... - [Data Modeling for the Lakehouse: What Changes](https://iceberglakehouse.com/posts/2026-02-dm-data-modeling-lakehouse/) (2026-02-18): Traditional data modeling assumed you controlled the database. You defined schemas up front, enforced foreign keys at write time, and optimized with indexes.... - [Semantic Layer vs. Data Catalog: Complementary, Not Competing](https://iceberglakehouse.com/posts/2026-02-sl-semantic-layer-vs-data-catalog/) (2026-02-18): "We already have a data catalog, so we don''t need a semantic layer. " This is one of the most common misconceptions in modern data architecture. - [Data Quality Is a Pipeline Problem, Not a Dashboard Problem](https://iceberglakehouse.com/posts/2026-02-debp-data-quality-first/) (2026-02-18): When an analyst finds null values in a revenue column, the typical response is to add a calculated field in the BI tool - `IF revenue IS NULL THEN 0`. - [Star Schema vs. Snowflake Schema: When to Use Each](https://iceberglakehouse.com/posts/2026-02-dm-star-schema-vs-snowflake/) (2026-02-18): Both star schemas and snowflake schemas are dimensional models. They both organize data into fact tables (measurable events) and dimension tables (context ab... - [Semantic Layer vs. Metrics Layer: What's the Difference?](https://iceberglakehouse.com/posts/2026-02-sl-semantic-layer-vs-metrics-layer/) (2026-02-18): Both terms appear in every modern data architecture diagram. They're used interchangeably in conference talks, Slack threads, and vendor marketing. And almos... - [How to Design Reliable Data Pipelines](https://iceberglakehouse.com/posts/2026-02-debp-design-data-pipelines/) (2026-02-18): Most pipeline failures aren't caused by bad code. They're caused by no architecture. A script that reads from an API, transforms JSON, and writes to a databa... - [Conceptual, Logical, and Physical Data Models Explained](https://iceberglakehouse.com/posts/2026-02-dm-types-of-data-models/) (2026-02-18): Most data teams jump straight from a stakeholder request to creating database tables. They skip the planning steps that prevent misalignment, redundancy, and... - [How to Build a Semantic Layer: A Step-by-Step Guide](https://iceberglakehouse.com/posts/2026-02-sl-how-to-build-semantic-layer/) (2026-02-18): Most teams start building a semantic layer the wrong way: they open their BI tool, create a few calculated fields, and call it done. Six months later, three ... - [How to Think Like a Data Engineer](https://iceberglakehouse.com/posts/2026-02-debp-think-like-data-engineer/) (2026-02-18): The median lifespan of a popular data tool is about three years. The tool you master today may be deprecated or replaced by the time your next project ships.... - [What Is Data Modeling? A Complete Guide](https://iceberglakehouse.com/posts/2026-02-dm-what-is-data-modeling/) (2026-02-18): Every database, data warehouse, and data lakehouse starts with the same question: how should this data be organized? Data modeling answers that question by c... - [What Is a Semantic Layer? A Complete Guide](https://iceberglakehouse.com/posts/2026-02-sl-what-is-a-semantic-layer/) (2026-02-18): Ask three teams in your company how they calculate "revenue" and you''ll get three answers. Sales counts bookings. Finance counts recognized revenue. - [A 2026 Introduction to Apache Iceberg](https://iceberglakehouse.com/posts/2026-02-intro-to-apache-iceberg/) (2026-02-13): An updated introduction to Apache Iceberg - [RAG Isn’t a Modeling Problem. It’s a Data Engineering Problem.](https://iceberglakehouse.com/posts/2026-01-rag-isnt-the-problem/) (2026-01-20): Why retrieval-augmented generation systems fail in enterprises - and what to do about it. - [Building Pangolin - My Holiday Break, an AI IDE, and a Lakehouse Catalog for the Curious](https://iceberglakehouse.com/posts/2026-01-the-story-of-pangolin-catalog/) (2026-01-15): A personal story of how I built Pangolin Catalog over a holiday break using an AI-powered IDE. - [dremioframe & iceberg - Pythonic interfaces for Dremio and Apache Iceberg](https://iceberglakehouse.com/posts/2025-12-dremioframe-and-iceframe/) (2025-12-05): Discover DremioFrame and IceFrame, two new Python libraries that simplify working with Dremio and Apache Iceberg. - [Introducing dremioframe - A Pythonic DataFrame Interface for Dremio](https://iceberglakehouse.com/posts/2025-11-introducing-dremioframe-dataframe-python-library/) (2025-11-29): Discover dremioframe, a new Python library that offers a DataFrame-like experience for interacting with Dremio's data lakehouse platform. - [Comprehensive Hands-on Walk Through of Dremio Cloud Next Gen (Hands-on with Free Trial)](https://iceberglakehouse.com/posts/2025-11-dremio-next-gen-cloud-tutorial/) (2025-11-12): Walkthrough with the new trial of the Dremio Cloud Platform - [An Exploration of the Commercial Iceberg Catalog Ecosystem](https://iceberglakehouse.com/posts/2025-10-exploring-commerical-apache-iceberg-catalogs/) (2025-10-21): Dive into the world of commercial Iceberg catalogs and discover how they enhance data lakehouse architectures for modern data engineering. - [Building a Universal Lakehouse Catalog - Beyond Iceberg Tables](https://iceberglakehouse.com/posts/2025-10-building-universal-lakehouse-catalog/) (2025-10-17): Exploring paths to a universal lakehouse catalog that supports multiple data formats and engines, building on Apache Iceberg's success. - [Intro to Apache Iceberg with Apache Polaris and Apache Spark](https://iceberglakehouse.com/posts/2025-10-intro-to-apache-iceberg-with-apache-polaris-and-apache-spark/) (2025-10-16): Learn how to leverage Apache Iceberg with Apache Polaris and Apache Spark to build scalable and efficient data lakehouses. - [Composable Analytics with Agents - Leveraging Virtual Datasets and the Semantic Layer](https://iceberglakehouse.com/posts/2025-09-composable-analytics-with-agents/) (2025-09-17): The power of Dremio's Semantic Layer for Agentic AI - [Unlocking the Power of Agentic AI with Apache Iceberg and Dremio](https://iceberglakehouse.com/posts/2025-09-agentic-ai-dremio-apache-iceberg/) (2025-09-05): Unlocking the Power of Agentic AI with Apache Iceberg and Dremio - [What is an API? And Why Data Architecture Depends on Them](https://iceberglakehouse.com/posts/2025-06-what-is-an-api/) (2025-06-23): Hive, Glue, Iceberg Rest and so many more! - [Decoding AWS EC2 Instance Type Names](https://iceberglakehouse.com/posts/2025-06-aws-instance-types/) (2025-06-18): understanding the name of different AWS Instance Types - [Introduction to Data Engineering Concepts | What is Data Engineering?](https://iceberglakehouse.com/posts/2025-05-intro-to-data-engineering-concepts-01/) (2025-05-02): Introduction to the terms in data engineering - [Introduction to Data Engineering Concepts | Understanding Data Sources and Ingestion](https://iceberglakehouse.com/posts/2025-05-intro-to-data-engineering-concepts-02/) (2025-05-02): Introduction to the terms in data engineering - [Introduction to Data Engineering Concepts | ETL vs ELT – Understanding Data Pipelines](https://iceberglakehouse.com/posts/2025-05-intro-to-data-engineering-concepts-03/) (2025-05-02): Introduction to the terms in data engineering - [Introduction to Data Engineering Concepts | Batch Processing Fundamentals](https://iceberglakehouse.com/posts/2025-05-intro-to-data-engineering-concepts-04/) (2025-05-02): Introduction to the terms in data engineering - [Introduction to Data Engineering Concepts | Streaming Data Fundamentals](https://iceberglakehouse.com/posts/2025-05-intro-to-data-engineering-concepts-05/) (2025-05-02): Introduction to the terms in data engineering - [Introduction to Data Engineering Concepts | Data Modeling Basics](https://iceberglakehouse.com/posts/2025-05-intro-to-data-engineering-concepts-06/) (2025-05-02): Introduction to the terms in data engineering - [Introduction to Data Engineering Concepts | Data Warehousing Fundamentals](https://iceberglakehouse.com/posts/2025-05-intro-to-data-engineering-concepts-07/) (2025-05-02): Introduction to the terms in data engineering - [Introduction to Data Engineering Concepts | Data Lakes Explained](https://iceberglakehouse.com/posts/2025-05-intro-to-data-engineering-concepts-08/) (2025-05-02): Introduction to the terms in data engineering - [Introduction to Data Engineering Concepts | Storage Formats and Compression](https://iceberglakehouse.com/posts/2025-05-intro-to-data-engineering-concepts-09/) (2025-05-02): Introduction to the terms in data engineering - [Introduction to Data Engineering Concepts | Data Quality and Validation](https://iceberglakehouse.com/posts/2025-05-intro-to-data-engineering-concepts-10/) (2025-05-02): Introduction to the terms in data engineering - [Introduction to Data Engineering Concepts | Metadata, Lineage, and Governance](https://iceberglakehouse.com/posts/2025-05-intro-to-data-engineering-concepts-11/) (2025-05-02): Introduction to the terms in data engineering - [Introduction to Data Engineering Concepts | Scheduling and Workflow Orchestration](https://iceberglakehouse.com/posts/2025-05-intro-to-data-engineering-concepts-12/) (2025-05-02): Introduction to the terms in data engineering - [Introduction to Data Engineering Concepts | Building Scalable Pipelines](https://iceberglakehouse.com/posts/2025-05-intro-to-data-engineering-concepts-13/) (2025-05-02): Introduction to the terms in data engineering - [Introduction to Data Engineering Concepts | DevOps for Data Engineering](https://iceberglakehouse.com/posts/2025-05-intro-to-data-engineering-concepts-14/) (2025-05-02): Introduction to the terms in data engineering - [Introduction to Data Engineering Concepts | Cloud Data Platforms and the Modern Stack](https://iceberglakehouse.com/posts/2025-05-intro-to-data-engineering-concepts-15/) (2025-05-02): Introduction to the terms in data engineering - [Introduction to Data Engineering Concepts | Data Lakehouse Architecture Explained](https://iceberglakehouse.com/posts/2025-05-intro-to-data-engineering-concepts-16/) (2025-05-02): Introduction to the terms in data engineering - [Introduction to Data Engineering Concepts | Apache Iceberg, Arrow, and Polaris](https://iceberglakehouse.com/posts/2025-05-intro-to-data-engineering-concepts-17/) (2025-05-02): Introduction to the terms in data engineering - [Introduction to Data Engineering Concepts | The Power of Dremio in the Modern Lakehouse](https://iceberglakehouse.com/posts/2025-05-intro-to-data-engineering-concepts-18/) (2025-05-02): Introduction to the terms in data engineering - [Using Helm with Kubernetes - A Guide to Helm Charts and Their Implementation](https://iceberglakehouse.com/posts/2025-02-using-helm-with-kubernetes/) (2025-02-19): A Guide on when to use Helm Charts for Kubernetes Deployment - [Intro to SQL using Apache Iceberg and Dremio](https://iceberglakehouse.com/posts/2024-11-intro-to-sql-with-dremio-and-apache-iceberg/) (2024-11-08): Intro to SQL using Apache Iceberg and Dremio - [Introduction to Cargo and cargo.toml](https://iceberglakehouse.com/posts/2024-11-rust-cargo-toml/) (2024-11-05): Getting Started with Cargo and cargo.toml - [Leveraging Python's Pattern Matching and Comprehensions for Data Analytics](https://iceberglakehouse.com/posts/2024-11-python-analytics-pattern-matching/) (2024-11-01): Using Features like Pattern Matching and Comprehensions for Data Analytics - [Data Modeling - Entities and Events](https://iceberglakehouse.com/posts/2024-10-data-modeling-events-entities/) (2024-10-30): How to Model Events and Entities - [Orchestrating Airflow DAGs with GitHub Actions - A Lightweight Approach to Data Curation Across Spark, Dremio, and Snowflake](https://iceberglakehouse.com/posts/2024-10-github-actions-dbt-airflow-data/) (2024-10-19): Advanced GitHub Actions for Data Engineering - [A Deep Dive Into GitHub Actions From Software Development to Data Engineering](https://iceberglakehouse.com/posts/2024-10-intro-to-github-actions/) (2024-10-19): Learning about GitHub Actions - [Deep Dive into Data Apps with Streamlit](https://iceberglakehouse.com/posts/2024-9-deep-dive-into-data-apps-with-streamlit/) (2024-09-22): Building a Deploying Data Apps Easily - [A Deep Dive into Docker Compose](https://iceberglakehouse.com/posts/2024-9-a-deep-dive-into-docker-compose/) (2024-09-21): A Comprehensive Guide to Docker Compose ## Articles: Governance and Security - [Guardrails for AI on Company Data](https://iceberglakehouse.com/posts/guardrails-ai-company-data/) (2026-09-10): The control surfaces that actually contain damage once an agent is fooled: identity, permissions, audit trails, and prompt injection at the query layer. - [How Iceberg Catalogs Hand Engines Storage Access](https://iceberglakehouse.com/posts/iceberg-vended-credentials/) (2026-09-10): Credential vending end to end: the wire protocol, scoped access on each cloud, remote signing, credential lifetime on long jobs, and failures that look like bugs. - [Deleting User Data From an Immutable Lakehouse: GDPR Hard Deletes on Iceberg](https://iceberglakehouse.com/posts/gdpr-hard-deletes-on-iceberg/) (2026-09-02): How to turn a logical delete on immutable Iceberg into a physical erasure across snapshots, versions, replicas, and downstream copies. - [Metadata Platforms in 2026: DataHub, OpenMetadata, Atlan, and Catalog Convergence](https://iceberglakehouse.com/posts/metadata-platforms-in-2026/) (2026-09-02): How the technical catalog and the metadata platform are converging in 2026, and how to arrange the two layers for a lakehouse. - [Governance-as-Code for the Lakehouse: Managing REST Catalog RBAC and Masking in Git](https://iceberglakehouse.com/posts/governance-as-code-lakehouse-rest-catalog/) (2026-08-25): Put REST catalog RBAC and masking in Git. How to review grants, apply them safely, and keep lakehouse access from drifting. - [Policy-Aware Lakehouse Telemetry: Building Auditable AI Records on Apache Iceberg for the EU AI Act Era](https://iceberglakehouse.com/posts/policy-aware-lakehouse-telemetry-eu-ai-act/) (2026-08-19): Policy-aware lakehouse telemetry for the EU AI Act era: build auditable, immutable, governed AI decision records on Apache Iceberg. - [Securing the Agentic Lakehouse Gateway: A Threat Model for Prompt Injection, Exfiltration, and the Firewall That Reads Sentences](https://iceberglakehouse.com/posts/securing-agentic-lakehouse-gateway/) (2026-08-19): A threat model for the agentic lakehouse gateway covering prompt injection, exfiltration, and the firewall that reads sentences. - [Policy Layer for Autonomous AI Data Agents](https://iceberglakehouse.com/posts/policy-layer-security-policy-enforcement-autonomous-multi-agent-systems/) (2026-07-13): An in-depth exploration of policy layer for autonomous ai data agents ## Articles: Other - [Where Lock-In Went](https://iceberglakehouse.com/posts/where-lock-in-went/) (2026-09-10): The format war ended and exit cost did not: where lock-in relocated after open tables won, how to measure it, and which costs are worth keeping down. - [The Filters We Build: Scams, Slop, and the Search for Signal, From Radio Ads to AI](https://iceberglakehouse.com/posts/cognitive-filters-media-evolution/) (2026-07-24): How every media era from radio to AI forces humanity to build new cognitive filters, the shield against scams and the sieve for finding signal in an ocean. - [Open Table Format Benchmarks: Why They Require Critical Evaluation](https://iceberglakehouse.com/posts/2026-05-22-open-table-format-benchmarks-guide/) (2026-05-22): An in-depth analysis of open table format benchmarks comparing Apache Iceberg, Delta Lake, and Apache Hudi, detailing the pitfalls of standard benchmarks. - [Apache Iceberg SCD Type 2 and CDC Patterns: Building Historical Lakehouse Tables](https://iceberglakehouse.com/posts/2026-05-22-apache-iceberg-scd-type-2-cdc-patterns/) (2026-05-22): A deep dive into implementing Slowly Changing Dimension Type 2 (SCD Type 2) patterns and Change Data Capture (CDC) pipelines on Apache Iceberg, using. - [Apache Iceberg Catalogs Explained: REST, Glue, Hive Metastore, Polaris, Nessie, and Snowflake](https://iceberglakehouse.com/posts/2026-05-22-apache-iceberg-catalogs-explained/) (2026-05-22): A deep dive into Apache Iceberg catalog architecture, comparing REST catalogs, AWS Glue, Project Nessie, Polaris, and Snowflake. - [Apache Iceberg with Spark: Create, MERGE, Upsert, and Evolve Tables End to End](https://iceberglakehouse.com/posts/2026-05-22-apache-iceberg-spark-dml-evolution/) (2026-05-22): A comprehensive developer guide to configuring Apache Spark with Apache Iceberg, executing transactional writes, and managing schema evolution. - [Building a Multicloud Agentic Lakehouse Reference Architecture](https://iceberglakehouse.com/posts/2026-05-22-multicloud-agentic-lakehouse-reference-architecture/) (2026-05-22): A reference architecture for building an open, multicloud Data Lakehouse optimized for AI Agents using Apache Polaris, Apache Iceberg, and Dremio. - [Common Misconceptions About Data Lakehouse and Apache Iceberg](https://iceberglakehouse.com/posts/2026-05-22-data-lakehouse-and-iceberg-misconceptions/) (2026-05-22): Addressing common search queries and reader confusion about Data Lakehouse architectures, Apache Iceberg catalogs, partitions, and lock-in. ## Apache Iceberg Knowledge Base - [Knowledge Base Index](https://iceberglakehouse.com/iceberg/) - [Amazon S3 Tables for Apache Iceberg](https://iceberglakehouse.com/iceberg/amazon-s3-tables/) - [Apache DataFusion](https://iceberglakehouse.com/iceberg/apache-datafusion/) - [Apache Polaris Catalog](https://iceberglakehouse.com/iceberg/apache-polaris-catalog/) - [Apache XTable Translations](https://iceberglakehouse.com/iceberg/apache-xtable-translations/) - [AWS Athena and Apache Iceberg](https://iceberglakehouse.com/iceberg/aws-athena-iceberg/) - [Amazon EMR and Apache Iceberg](https://iceberglakehouse.com/iceberg/aws-emr-iceberg/) - [AWS Glue Catalog for Apache Iceberg](https://iceberglakehouse.com/iceberg/aws-glue-catalog/) - [Microsoft Fabric and Apache Iceberg](https://iceberglakehouse.com/iceberg/azure-fabric-iceberg/) - [Bauplan](https://iceberglakehouse.com/iceberg/bauplan/) - [BigQuery and Apache Iceberg](https://iceberglakehouse.com/iceberg/bigquery-apache-iceberg/) - [Catalog Federation](https://iceberglakehouse.com/iceberg/catalog-federation/) - [Catalog Namespaces](https://iceberglakehouse.com/iceberg/catalog-namespaces/) - [CDC Log Ingestion Pipelines](https://iceberglakehouse.com/iceberg/cdc-log-ingestion-pipelines/) - [ClickHouse](https://iceberglakehouse.com/iceberg/clickhouse/) - [Columnar Memory Layouts](https://iceberglakehouse.com/iceberg/columnar-memory-layouts/) - [Data Lakehouse](https://iceberglakehouse.com/iceberg/data-lakehouse/) - [Data Lineage Tracking](https://iceberglakehouse.com/iceberg/data-lineage-tracking/) - [Databricks and Apache Iceberg](https://iceberglakehouse.com/iceberg/databricks-iceberg/) - [dbt and Apache Iceberg](https://iceberglakehouse.com/iceberg/dbt-apache-iceberg/) - [Debezium CDC Engines](https://iceberglakehouse.com/iceberg/debezium-cdc-engines/) - [Decoupled Compute and Storage](https://iceberglakehouse.com/iceberg/decoupled-compute-and-storage/) - [Delta Lake UniForm Metadata](https://iceberglakehouse.com/iceberg/delta-lake-uniform-metadata/) - [Apache Doris and Apache Iceberg](https://iceberglakehouse.com/iceberg/doris-apache-iceberg/) - [Dremio Acceleration Engine](https://iceberglakehouse.com/iceberg/dremio-acceleration-engine/) - [Dremio Aggregation Reflections](https://iceberglakehouse.com/iceberg/dremio-aggregation-reflections/) - [Dremio and Apache Iceberg](https://iceberglakehouse.com/iceberg/dremio-apache-iceberg/) - [Dremio Arrow Flight SQL](https://iceberglakehouse.com/iceberg/dremio-arrow-flight-sql/) - [Dremio Column-Level Masking](https://iceberglakehouse.com/iceberg/dremio-column-level-masking/) - [Dremio Columnar Cloud Cache (C3)](https://iceberglakehouse.com/iceberg/dremio-columnar-cloud-cache-c3/) - [Dremio Coordinator Node](https://iceberglakehouse.com/iceberg/dremio-coordinator-node/) - [Dremio Data Reflections Matching](https://iceberglakehouse.com/iceberg/dremio-data-reflections-matching/) - [Dremio Engine Auto-scaling](https://iceberglakehouse.com/iceberg/dremio-engine-auto-scaling/) - [Dremio External Queries](https://iceberglakehouse.com/iceberg/dremio-external-queries/) - [Dremio Iceberg Metadata Sync](https://iceberglakehouse.com/iceberg/dremio-iceberg-metadata-sync/) - [Dremio Join Co-segmentation](https://iceberglakehouse.com/iceberg/dremio-join-co-segmentation/) - [Dremio LDAP Integration](https://iceberglakehouse.com/iceberg/dremio-ldap-integration/) - [Dremio Metadata Caching](https://iceberglakehouse.com/iceberg/dremio-metadata-caching/) - [Dremio Parquet Vectorized Reader](https://iceberglakehouse.com/iceberg/dremio-parquet-vectorized-reader/) - [Dremio Physical Datasets (PDS)](https://iceberglakehouse.com/iceberg/dremio-physical-datasets-pds/) - [Dremio Raw Reflections](https://iceberglakehouse.com/iceberg/dremio-raw-reflections/) - [Dremio Reflections](https://iceberglakehouse.com/iceberg/dremio-reflections/) - [Dremio Row-Level Security (RLS)](https://iceberglakehouse.com/iceberg/dremio-row-level-security-rls/) - [Dremio Sabot Engine](https://iceberglakehouse.com/iceberg/dremio-sabot-engine/) - [Dremio Spaces](https://iceberglakehouse.com/iceberg/dremio-spaces/) - [Dremio SQL Runner](https://iceberglakehouse.com/iceberg/dremio-sql-runner/) - [Dremio User Defined Functions (UDFs)](https://iceberglakehouse.com/iceberg/dremio-user-defined-functions-udfs/) - [Dremio Virtual Datasets (VDS)](https://iceberglakehouse.com/iceberg/dremio-virtual-datasets-vds/) - [DuckDB and Apache Iceberg](https://iceberglakehouse.com/iceberg/duckdb-apache-iceberg/) - [Dynamic Filter Pushdown](https://iceberglakehouse.com/iceberg/dynamic-filter-pushdown/) - [Apache Flink and Apache Iceberg](https://iceberglakehouse.com/iceberg/flink-apache-iceberg/) - [Glue Catalog IAM Policies](https://iceberglakehouse.com/iceberg/glue-catalog-iam-policies/) - [Glue Catalog Lake Formation](https://iceberglakehouse.com/iceberg/glue-catalog-lake-formation/) - [Google Cloud BigLake](https://iceberglakehouse.com/iceberg/google-cloud-biglake/) - [Google Cloud and Apache Iceberg](https://iceberglakehouse.com/iceberg/google-cloud-iceberg/) - [Apache Gravitino](https://iceberglakehouse.com/iceberg/gravitino-catalog/) - [Hive and Apache Iceberg](https://iceberglakehouse.com/iceberg/hive-apache-iceberg/) - [Iceberg Access Control Patterns](https://iceberglakehouse.com/iceberg/iceberg-access-control/) - [ACID Transactions in Apache Iceberg](https://iceberglakehouse.com/iceberg/iceberg-acid-transactions/) - [Agentic Lakehouse](https://iceberglakehouse.com/iceberg/iceberg-agentic-lakehouse/) - [Iceberg AI Readiness](https://iceberglakehouse.com/iceberg/iceberg-ai-readiness/) - [Iceberg AI Semantic Layer](https://iceberglakehouse.com/iceberg/iceberg-ai-semantic-layer/) - [Apache Airflow and Apache Iceberg](https://iceberglakehouse.com/iceberg/iceberg-apache-airflow/) - [Iceberg Apache Arrow Flight](https://iceberglakehouse.com/iceberg/iceberg-arrow-flight/) - [Iceberg Audit Logging](https://iceberglakehouse.com/iceberg/iceberg-audit-logging/) - [Iceberg Avro Metadata Format](https://iceberglakehouse.com/iceberg/iceberg-avro-format/) - [Iceberg Bin-Packing Compaction](https://iceberglakehouse.com/iceberg/iceberg-bin-packing-compaction/) - [Iceberg Bloom Filters](https://iceberglakehouse.com/iceberg/iceberg-bloom-filters/) - [Iceberg Branching and Tagging](https://iceberglakehouse.com/iceberg/iceberg-branching-tagging/) - [Iceberg Bucket Partition Transform](https://iceberglakehouse.com/iceberg/iceberg-bucket-partition-transform/) - [Iceberg Catalog Migration](https://iceberglakehouse.com/iceberg/iceberg-catalog-migration/) - [What is an Iceberg Catalog?](https://iceberglakehouse.com/iceberg/iceberg-catalog/) - [Iceberg CDC (Change Data Capture)](https://iceberglakehouse.com/iceberg/iceberg-cdc/) - [Iceberg Table Clustering](https://iceberglakehouse.com/iceberg/iceberg-clustering/) - [Iceberg Column Mapping](https://iceberglakehouse.com/iceberg/iceberg-column-mapping/) - [Iceberg Column Projection](https://iceberglakehouse.com/iceberg/iceberg-column-projection/) - [Iceberg Table Compaction](https://iceberglakehouse.com/iceberg/iceberg-compaction/) - [Iceberg Concurrent Write Handling](https://iceberglakehouse.com/iceberg/iceberg-concurrent-writes/) - [Copy-on-Write (CoW) in Iceberg](https://iceberglakehouse.com/iceberg/iceberg-copy-on-write/) - [Iceberg Cost Optimization](https://iceberglakehouse.com/iceberg/iceberg-cost-optimization/) - [Iceberg Data Files](https://iceberglakehouse.com/iceberg/iceberg-data-files/) - [Iceberg Data Lineage](https://iceberglakehouse.com/iceberg/iceberg-data-lineage/) - [Iceberg Data Masking](https://iceberglakehouse.com/iceberg/iceberg-data-masking/) - [Iceberg Data Mesh Architecture](https://iceberglakehouse.com/iceberg/iceberg-data-mesh/) - [Iceberg Data Skipping](https://iceberglakehouse.com/iceberg/iceberg-data-skipping/) - [Iceberg Date/Time Partition Transforms](https://iceberglakehouse.com/iceberg/iceberg-date-time-partition-transforms/) - [Iceberg Decimal Type Widening](https://iceberglakehouse.com/iceberg/iceberg-decimal-type-widening/) - [Iceberg Delete Files](https://iceberglakehouse.com/iceberg/iceberg-delete-files/) - [Iceberg Deletion Vectors](https://iceberglakehouse.com/iceberg/iceberg-deletion-vectors/) - [Iceberg DynamoDB Catalog](https://iceberglakehouse.com/iceberg/iceberg-dynamodb-catalog/) - [Iceberg Encryption](https://iceberglakehouse.com/iceberg/iceberg-encryption/) - [Iceberg Equality Deletes](https://iceberglakehouse.com/iceberg/iceberg-equality-deletes/) - [Expire Snapshots in Apache Iceberg](https://iceberglakehouse.com/iceberg/iceberg-expire-snapshots/) - [Iceberg Feature Store](https://iceberglakehouse.com/iceberg/iceberg-feature-store/) - [Iceberg File Content Type](https://iceberglakehouse.com/iceberg/iceberg-file-content-type/) - [Iceberg FileIO API](https://iceberglakehouse.com/iceberg/iceberg-file-io/) - [Iceberg File Path Spec](https://iceberglakehouse.com/iceberg/iceberg-file-path-spec/) - [Hidden Partitioning in Apache Iceberg](https://iceberglakehouse.com/iceberg/iceberg-hidden-partitioning/) - [Iceberg Hive Catalog Lock Manager](https://iceberglakehouse.com/iceberg/iceberg-hive-catalog-lock-manager/) - [Hive Metastore Catalog for Iceberg](https://iceberglakehouse.com/iceberg/iceberg-hive-metastore/) - [Iceberg Identity Partition Transform](https://iceberglakehouse.com/iceberg/iceberg-identity-partition-transform/) - [Iceberg Incremental Reads](https://iceberglakehouse.com/iceberg/iceberg-incremental-read/) - [Iceberg JDBC Catalog Locks](https://iceberglakehouse.com/iceberg/iceberg-jdbc-catalog-locks/) - [Iceberg JDBC Catalog](https://iceberglakehouse.com/iceberg/iceberg-jdbc-catalog/) - [Iceberg Lakehouse Federation](https://iceberglakehouse.com/iceberg/iceberg-lakehouse-federation/) - [Iceberg LLM Grounding and RAG for Structured Data](https://iceberglakehouse.com/iceberg/iceberg-llm-grounding/) - [Iceberg Lock Manager](https://iceberglakehouse.com/iceberg/iceberg-lock-manager/) - [Iceberg Maintenance Scheduling](https://iceberglakehouse.com/iceberg/iceberg-maintenance-scheduling/) - [Iceberg Manifest Entry Schema](https://iceberglakehouse.com/iceberg/iceberg-manifest-entry-schema/) - [Iceberg Manifest Entry Status](https://iceberglakehouse.com/iceberg/iceberg-manifest-entry-status/) - [Iceberg Manifest File](https://iceberglakehouse.com/iceberg/iceberg-manifest-file/) - [Iceberg Manifest List Schema](https://iceberglakehouse.com/iceberg/iceberg-manifest-list-schema/) - [Iceberg Manifest List](https://iceberglakehouse.com/iceberg/iceberg-manifest-list/) - [Iceberg Manifest Merging](https://iceberglakehouse.com/iceberg/iceberg-manifest-merging/) - [MCP and Apache Iceberg](https://iceberglakehouse.com/iceberg/iceberg-mcp/) - [Medallion Architecture with Apache Iceberg](https://iceberglakehouse.com/iceberg/iceberg-medallion-architecture/) - [Merge-on-Read (MoR) in Iceberg](https://iceberglakehouse.com/iceberg/iceberg-merge-on-read/) - [Iceberg Metadata File](https://iceberglakehouse.com/iceberg/iceberg-metadata-file/) - [Iceberg Metadata Pruning](https://iceberglakehouse.com/iceberg/iceberg-metadata-pruning/) - [Iceberg Metadata Table Files](https://iceberglakehouse.com/iceberg/iceberg-metadata-table-files/) - [Iceberg Metadata Table History](https://iceberglakehouse.com/iceberg/iceberg-metadata-table-history/) - [Iceberg Metadata Table Snapshots](https://iceberglakehouse.com/iceberg/iceberg-metadata-table-snapshots/) - [Iceberg Metrics Mode](https://iceberglakehouse.com/iceberg/iceberg-metrics-mode/) - [Iceberg Table Migration from Hive](https://iceberglakehouse.com/iceberg/iceberg-migration-hive/) - [Iceberg Multi-Catalog Architecture](https://iceberglakehouse.com/iceberg/iceberg-multi-catalog/) - [Iceberg Multi-Tenancy Patterns](https://iceberglakehouse.com/iceberg/iceberg-multi-tenant/) - [Iceberg Natural Language Analytics](https://iceberglakehouse.com/iceberg/iceberg-natural-language/) - [Iceberg Nested Type System](https://iceberglakehouse.com/iceberg/iceberg-nested-type-system/) - [Iceberg Open Table Format vs. Delta Lake vs. Apache Hudi](https://iceberglakehouse.com/iceberg/iceberg-open-table-format/) - [Iceberg Optimistic Concurrency Control (OCC)](https://iceberglakehouse.com/iceberg/iceberg-optimistic-concurrency-control-occ/) - [Apache Iceberg ORC Format](https://iceberglakehouse.com/iceberg/iceberg-orc-format/) - [Iceberg Orphan Files Penalty](https://iceberglakehouse.com/iceberg/iceberg-orphan-files-penalty/) - [Iceberg Orphan Files](https://iceberglakehouse.com/iceberg/iceberg-orphan-files/) - [Iceberg Parent Snapshot ID](https://iceberglakehouse.com/iceberg/iceberg-parent-snapshot-id/) - [Apache Parquet and Iceberg](https://iceberglakehouse.com/iceberg/iceberg-parquet/) - [Partition Evolution in Apache Iceberg](https://iceberglakehouse.com/iceberg/iceberg-partition-evolution/) - [Iceberg Partition-Level Compaction](https://iceberglakehouse.com/iceberg/iceberg-partition-level-compaction/) - [Iceberg Performance Tuning Guide](https://iceberglakehouse.com/iceberg/iceberg-performance-tuning/) - [Iceberg Positional Deletes](https://iceberglakehouse.com/iceberg/iceberg-positional-deletes/) - [Iceberg Predicate Pushdown](https://iceberglakehouse.com/iceberg/iceberg-predicate-pushdown/) - [Iceberg Puffin Files](https://iceberglakehouse.com/iceberg/iceberg-puffin-files/) - [Iceberg REST Catalog API Reference](https://iceberglakehouse.com/iceberg/iceberg-rest-catalog-api/) - [Iceberg REST Catalog](https://iceberglakehouse.com/iceberg/iceberg-rest-catalog/) - [Iceberg Rewrite Manifests](https://iceberglakehouse.com/iceberg/iceberg-rewrite-manifests/) - [Iceberg Rollback Snapshot Procedures](https://iceberglakehouse.com/iceberg/iceberg-rollback-snapshot-procedures/) - [Iceberg Table Rollback](https://iceberglakehouse.com/iceberg/iceberg-rollback/) - [Row-Level Deletes in Apache Iceberg](https://iceberglakehouse.com/iceberg/iceberg-row-level-deletes/) - [Schema Evolution in Apache Iceberg](https://iceberglakehouse.com/iceberg/iceberg-schema-evolution/) - [Iceberg Sequence Number Inheritance](https://iceberglakehouse.com/iceberg/iceberg-sequence-number-inheritance/) - [Iceberg Sequence Number](https://iceberglakehouse.com/iceberg/iceberg-sequence-number/) - [Small File Problem in Apache Iceberg](https://iceberglakehouse.com/iceberg/iceberg-small-file-problem/) - [Iceberg Snapshot Expiration Age](https://iceberglakehouse.com/iceberg/iceberg-snapshot-expiration-age/) - [Iceberg Snapshot References](https://iceberglakehouse.com/iceberg/iceberg-snapshot-references/) - [Iceberg Snapshot Summary](https://iceberglakehouse.com/iceberg/iceberg-snapshot-summary/) - [Apache Iceberg Snapshot](https://iceberglakehouse.com/iceberg/iceberg-snapshot/) - [Iceberg Sort-Based Compaction](https://iceberglakehouse.com/iceberg/iceberg-sort-based-compaction/) - [Iceberg Sort Order](https://iceberglakehouse.com/iceberg/iceberg-sort-order/) - [Iceberg Spark Procedure add_files](https://iceberglakehouse.com/iceberg/iceberg-spark-procedure-add-files/) - [Iceberg Spark Procedure expire_snapshots](https://iceberglakehouse.com/iceberg/iceberg-spark-procedure-expire-snapshots/) - [Iceberg Spark Procedure register_table](https://iceberglakehouse.com/iceberg/iceberg-spark-procedure-register-table/) - [Iceberg Spark Procedure remove_orphan_files](https://iceberglakehouse.com/iceberg/iceberg-spark-procedure-remove-orphan-files/) - [Iceberg Spark Procedure rewrite_data_files](https://iceberglakehouse.com/iceberg/iceberg-spark-procedure-rewrite-data-files/) - [Iceberg Spark Procedure rewrite_manifests](https://iceberglakehouse.com/iceberg/iceberg-spark-procedure-rewrite-manifests/) - [Iceberg Spark Procedure rewrite_position_deletes](https://iceberglakehouse.com/iceberg/iceberg-spark-procedure-rewrite-position-deletes/) - [Apache Iceberg Spec v1 vs v2](https://iceberglakehouse.com/iceberg/iceberg-spec-v1-vs-v2/) - [Iceberg Spec V3 File Encryption](https://iceberglakehouse.com/iceberg/iceberg-spec-v3-file-encryption/) - [Iceberg Spec V3 Object-Store Storage Layout](https://iceberglakehouse.com/iceberg/iceberg-spec-v3-object-store-storage-layout/) - [Apache Iceberg Spec v3](https://iceberglakehouse.com/iceberg/iceberg-spec-v3/) - [Apache Iceberg Spec v4 (Current State)](https://iceberglakehouse.com/iceberg/iceberg-spec-v4/) - [Iceberg Table Statistics (Puffin)](https://iceberglakehouse.com/iceberg/iceberg-statistics/) - [Iceberg Streaming Ingestion](https://iceberglakehouse.com/iceberg/iceberg-streaming/) - [Iceberg Table Design Best Practices](https://iceberglakehouse.com/iceberg/iceberg-table-design/) - [Apache Iceberg Table Format](https://iceberglakehouse.com/iceberg/iceberg-table-format/) - [Iceberg Table Metadata Schema](https://iceberglakehouse.com/iceberg/iceberg-table-metadata-schema/) - [Iceberg Table Properties](https://iceberglakehouse.com/iceberg/iceberg-table-properties/) - [Time Travel in Apache Iceberg](https://iceberglakehouse.com/iceberg/iceberg-time-travel/) - [Iceberg Truncate Partition Transform](https://iceberglakehouse.com/iceberg/iceberg-truncate-partition-transform/) - [Iceberg Upsert (MERGE INTO)](https://iceberglakehouse.com/iceberg/iceberg-upsert/) - [Iceberg Views](https://iceberglakehouse.com/iceberg/iceberg-views/) - [Apache Iceberg vs Apache Hudi](https://iceberglakehouse.com/iceberg/iceberg-vs-apache-hudi/) - [Apache Iceberg vs Delta Lake](https://iceberglakehouse.com/iceberg/iceberg-vs-delta-lake/) - [Write-Audit-Publish (WAP) Pattern](https://iceberglakehouse.com/iceberg/iceberg-wap-pattern/) - [Iceberg Write Distribution Modes](https://iceberglakehouse.com/iceberg/iceberg-write-distribution/) - [Iceberg Z-Order Compaction](https://iceberglakehouse.com/iceberg/iceberg-z-order-compaction/) - [Z-Order Clustering in Apache Iceberg](https://iceberglakehouse.com/iceberg/iceberg-zorder/) - [Apache Kafka and Apache Iceberg](https://iceberglakehouse.com/iceberg/kafka-apache-iceberg/) - [LangChain and Apache Iceberg](https://iceberglakehouse.com/iceberg/langchain-iceberg/) - [Microsoft Fabric OneLake](https://iceberglakehouse.com/iceberg/microsoft-fabric-onelake/) - [Modern Data Stack (MDS)](https://iceberglakehouse.com/iceberg/modern-data-stack-mds/) - [Nessie Git-like Branching](https://iceberglakehouse.com/iceberg/nessie-git-like-branching/) - [Nessie Merging](https://iceberglakehouse.com/iceberg/nessie-merging/) - [Nessie Tagging](https://iceberglakehouse.com/iceberg/nessie-tagging/) - [Object Storage Prefix Hashing](https://iceberglakehouse.com/iceberg/object-storage-prefix-hashing/) - [Open Table Format Comparison (Iceberg, Delta Lake, Hudi, Paimon)](https://iceberglakehouse.com/iceberg/open-table-format-comparison/) - [Polaris Catalog Sharing](https://iceberglakehouse.com/iceberg/polaris-catalog-sharing/) - [Polaris RBAC Model](https://iceberglakehouse.com/iceberg/polaris-rbac-model/) - [Polaris Service Principals](https://iceberglakehouse.com/iceberg/polaris-service-principals/) - [Presto and Apache Iceberg](https://iceberglakehouse.com/iceberg/presto-apache-iceberg/) - [Project Nessie](https://iceberglakehouse.com/iceberg/project-nessie/) - [PuppyGraph](https://iceberglakehouse.com/iceberg/puppygraph/) - [PyIceberg: Python Library for Apache Iceberg](https://iceberglakehouse.com/iceberg/pyiceberg/) - [Read Amplification](https://iceberglakehouse.com/iceberg/read-amplification/) - [REST Catalog Credential Vending](https://iceberglakehouse.com/iceberg/rest-catalog-credential-vending/) - [REST Catalog OAuth2 Token Flow](https://iceberglakehouse.com/iceberg/rest-catalog-oauth2-token-flow/) - [Single Source of Truth (SSOT)](https://iceberglakehouse.com/iceberg/single-source-of-truth-ssot/) - [Snowflake External Table Catalog Sync](https://iceberglakehouse.com/iceberg/snowflake-external-table-catalog-sync/) - [Snowflake Iceberg Tables](https://iceberglakehouse.com/iceberg/snowflake-iceberg-tables/) - [Snowflake Managed Iceberg Tables](https://iceberglakehouse.com/iceberg/snowflake-managed-iceberg-tables/) - [Snowflake Open Catalog](https://iceberglakehouse.com/iceberg/snowflake-open-catalog/) - [Space Amplification](https://iceberglakehouse.com/iceberg/space-amplification/) - [Apache Spark and Apache Iceberg](https://iceberglakehouse.com/iceberg/spark-apache-iceberg/) - [Spice.ai](https://iceberglakehouse.com/iceberg/spiceai/) - [Split Planning Loops](https://iceberglakehouse.com/iceberg/split-planning-loops/) - [StarRocks and Apache Iceberg](https://iceberglakehouse.com/iceberg/starrocks-apache-iceberg/) - [Apache Superset and Apache Iceberg](https://iceberglakehouse.com/iceberg/superset-apache-iceberg/) - [Time To First Byte (TTFB)](https://iceberglakehouse.com/iceberg/time-to-first-byte-ttfb/) - [Trino and Apache Iceberg](https://iceberglakehouse.com/iceberg/trino-apache-iceberg/) - [Unity Catalog Delta-Iceberg Compatibility](https://iceberglakehouse.com/iceberg/unity-catalog-delta-iceberg-compatibility/) - [VeloDB](https://iceberglakehouse.com/iceberg/velodb/) - [Vortex File Format](https://iceberglakehouse.com/iceberg/vortex-file-format/) - [What is Apache Iceberg?](https://iceberglakehouse.com/iceberg/what-is-apache-iceberg/) - [Write Amplification](https://iceberglakehouse.com/iceberg/write-amplification/) - [Zero-Copy Cloning](https://iceberglakehouse.com/iceberg/zero-copy-cloning/) ## Events - [Agentic Lakehouse Events](https://luma.com/agenticlakehouse): global meetups and webinars on agentic analytics - [Data Lakehouse Hub Events](https://luma.com/DataLakehouseHub): global lakehouse meetups, linkups and webinars ## Community - [Data Lakehouse Hub Slack](https://join.slack.com/t/thedatalakehousehub/shared_invite/zt-274yc8sza-mI2zhCW8LGkOh1uxuf8T5Q): practitioner community for lakehouse architecture - [Data Events Slack](https://join.slack.com/t/data-events/shared_invite/zt-38vgrooy9-U9ral_gr3NAz_Siih1QwmQ): announcements for data conferences and meetups - [Data & Tech Slack](https://join.slack.com/t/datatechcommunity/shared_invite/zt-12xrk4qmd-y~6jUFFd7kdaLhgLURKwoA): broader data and technology community - [r/datalakehouseandai](https://www.reddit.com/r/datalakehouseandai/): subreddit for data lakehouse and AI discussion - [Data Lakehouse Hub on LinkedIn](https://www.linkedin.com/company/data-lakehouse-hub/): company page for the Data Lakehouse Hub - [Alex Merced Tech on YouTube](https://www.youtube.com/@AlexMercedCoder): software development and engineering channel - [Alex Merced Data & AI on YouTube](https://www.youtube.com/@alexmerceddata): data lakehouse and AI channel