Lakehouse and table formats

Object storage gives you cheap, durable, effectively infinite files. What it does not give you is a table. A directory of Parquet files has no commit protocol - a writer that crashes halfway leaves a torn state visible to readers - no schema tracking beyond whatever the files happen to contain, and no way to ask “what did this look like yesterday”. An open table format adds the metadata layer that turns those files into a table a database would recognise: a defined schema, tracked partitions, atomic commits, and versioned snapshots.

Parquet: the file layer

Apache Parquet is the columnar file format the whole stack rests on. Columns are stored contiguously, so a query selecting three of two hundred columns reads only what it needs - which is why scan-heavy analytics lives on Parquet rather than row formats. Each file carries its schema, encoding, and statistics (min/max per column chunk) that let engines skip chunks that cannot contain matching rows. Parquet is a file format, not a table format: it describes how bytes are laid out, not how a collection of files behaves as one table.

Iceberg: the table layer

Apache Iceberg defines the metadata around the files:

• Snapshots. Every commit produces a new snapshot - a manifest list pointing at the data files that make up the table at that instant. Readers see a consistent snapshot; writers commit atomically by swapping which snapshot is current. A bad load is rolled back by pointing the table at the previous snapshot, not by deleting files.
• Schema evolution. Columns are tracked by id, not by name or position, so renaming, reordering, adding, and dropping columns are metadata operations - no file rewrites, and old files remain readable with the new schema.
• Partition evolution. The partition spec is versioned too: a table partitioned by day can switch to hourly partitioning going forward without rewriting history - old files stay partitioned by day, new files by hour, and queries plan correctly across both.
• Time travel and rollback. Snapshots are retained for a configured window, so SELECT ... AS OF reads a past version and ROLLBACK restores one - the audit and recovery properties a plain file directory cannot offer.
• Hidden partitioning. Query writers filter on the logical column (event_time) and Iceberg maps it to the physical partition transform - no requirement to spell the partition columns in every query.

Iceberg versus Delta Lake versus Hudi

The honest difference is governance of the metadata: Iceberg was designed as a neutral specification so any engine - Spark, Trino, Flink, Snowflake, DuckDB - reads the same table without a vendor’s format in the middle. Delta’s transaction log is equally capable technically, but the ecosystem around it is Databricks-shaped. If the table will be read by engines you have not chosen yet, Iceberg’s neutrality is the point; if you are all-in on Databricks, Delta is the path of least resistance.

The trade-offs nobody mentions

• Small-file problem. Streaming or frequent micro-batches produce thousands of tiny files; every query then pays for listing and opening them. Compaction is a maintenance job you run - rewrite_data_files in Iceberg, OPTIMIZE in Delta - not something the format does for you.
• Catalog dependency. The snapshot pointer lives in a Catalog implementation (Hive Metastore, AWS Glue, REST, Nessie). The table is only as available and as governed as that Catalog - a secondary piece of infrastructure that decides who can see and write the table.
• Commit conflicts are real. Two writers appending concurrently must reconcile; the formats do optimistic concurrency, which means retry storms under heavy contention. One writer per table, or partitioned write ownership, is the operational norm.
• Spec version is a compatibility boundary. Iceberg v3 features require engines that advertise v3 support; writing v3 metadata into a table some engine still reads as v2 is how tables become unreadable to part of the stack.

Failure modes and diagnostics

• Slow queries with high file counts. files / manifests metadata tables show file count per partition - compaction when the small-file ratio climbs.
• Readers seeing stale or partial data. Almost always a Catalog consistency or cache issue, or a reader on an old snapshot; check the current snapshot id the engine reports against the Catalog’s.
• Schema mismatch after evolution. A writer on a newer spec or an engine that does not honour the column-id model - the symptom is reads that return nulls or wrong columns for rows written before the change.
• Orphaned files and growing storage. Deletes and rollbacks leave data files the table no longer references; remove_orphan_files / expire_snapshots reclaim them - they do not clean themselves up.

What this is not for

A table format is not a database: no indexes, no OLTP point lookups, no sub-second row updates. If the workload is row-by-row transactions, that is an operational database’s job. And if the data lives in one engine with one team, the format’s neutrality is a feature you pay for in operational overhead and do not use - a managed warehouse’s native storage may be the simpler answer.