DuckLake is the new Delta Lake
A working claim, not a slogan. Delta Lake made open ACID lakehouses mainstream on object storage. DuckLake does the same job with a SQL catalog instead of a log of small metadata files — and leaves governance to you. Sequel to Field Note 04.
Every few years the industry renames the same stack and pretends the rename is a product. “Lakehouse” was one of those renames. Delta Lake was different: it made a concrete contract — ACID tables on Parquet, time travel, schema evolution — and let engines share the same files. DuckLake, from the DuckDB team, is the next concrete contract worth taking seriously. Not because it is louder. Because it stores the catalog as ordinary SQL tables in Postgres (or DuckDB, or SQLite), keeps the data as Parquet, and refuses to ship a Unity-style control plane as the format. That is either the new Delta — or a half-finished one. Both statements can be true at once.
Delta Lake made the open lakehouse real. DuckLake is the same job with a SQL table for a catalog — and a governance hole you still design yourself.
— working claimWhat “new Delta” is allowed to mean
If “new Delta” means “replace Databricks tomorrow,” the claim is false. If it means “the open table format that will absorb the next wave of mid-market lakehouses the way Delta absorbed the last,” the claim is a serious design argument. The rest of this note is that argument, with the limits left in.
Not a rebrand. Not a vendor. Not a governance product.
DuckLake is an open table format specification plus a DuckDB extension. The catalog is a SQL database you already know how to run. The data is Parquet. The format does not ship RBAC, row filters, or column masks as first-class features.
Allowed meaning
Same architectural job as Delta
ACID tables on object storage. Snapshots. Time travel. Schema evolution without rewriting every file. Multi-writer coordination through a catalog. That is what Delta industrialised for the open lakehouse era.
Disallowed meaning
Same product surface as Delta + UC
Unity Catalog’s RBAC, lineage, and multi-engine governance are not DuckLake features. Official docs and maintainers treat fine-grained access control as out of scope for the format. That gap is structural, not temporary marketing lag.
The problem Delta Lake actually fixed
Before Delta, “data lake” often meant a bucket of files with no reliable transactions. Concurrent writers raced. Readers saw half-written partitions. Time travel was a folder naming convention. Delta’s answer was a transaction log: commits as ordered records, data still in Parquet (and related columnar forms), engines reading the log to know what is current. That contract scaled across Spark and, later, a wider engine surface — and became the default mental model for an open lakehouse.
Contract 01
ACID on object storage
Commits are atomic. Readers do not see partial writes. Multiple writers coordinate through the log, not through hope.
Contract 02
Time travel as a product feature
Query as-of a version. Roll back. Audit what changed. The lake stops being a write-only dumping ground.
Contract 03
Schema evolution without panic
Add columns, rename carefully, evolve the table without rewriting history for every small change.
Contract 04
Open files, managed metadata
Data stays in open formats. The metadata protocol is the product. Governance layers (Unity Catalog and peers) then sat on top of that protocol for enterprises.
DuckLake’s move: the catalog is a database
DuckLake keeps the Delta-era split — data files separate from bookkeeping — but refuses the small-file metadata tree as the primary design. Snapshots, schemas, columns, data files, delete files, partition values, and stats live as rows in a catalog database. v1.0 defines 28 catalog tables. Validity is tracked with begin/end snapshot intervals. Concurrent writers race on a monotonic snapshot id; the extension retries when there is no logical conflict. That is not a REST catalog with a SQL UI. It is SQL as the catalog.
Log files vs rows
Both designs put Parquet (or equivalent) on object storage. The disagreement is where truth lives and how expensive it is to ask “which files are current?”
Delta / Iceberg style
Metadata as files on the lake
Transaction logs, manifests, and metadata trees on object storage. Avoids requiring a database. Pays for many small GETs and a separate catalog service when you want multi-user governance.
DuckLake style
Metadata as SQL tables
Postgres (preferred multi-user), DuckDB, or SQLite for the catalog. One SELECT replaces a tree walk. File pruning uses min/max stats stored as rows. The catalog is inspectable with ordinary SQL tools.
Where DuckLake is Delta’s peer, and where it is not
A category claim needs a table, not a mood. The comparison below is about the open table format job — not about Databricks workspaces, Photon, or Spark clusters. Those are compute products that often sit on top of Delta. DuckDB sits on top of DuckLake the same way Spark sits on top of Delta. Confusing the format with the compute is how the bill stays opaque.
| Dimension | Delta Lake (open format + typical enterprise stack) | DuckLake (open format + typical mid-market stack) |
|---|---|---|
| Data files | Parquet (and related columnar) on object storage | Parquet on any filesystem DuckDB can reach (local, S3, GCS, Azure, R2, etc.) |
| Catalog | Transaction log + metastore / Unity Catalog / REST catalogs | 28 versioned SQL tables in Postgres, DuckDB, or SQLite |
| ACID / snapshots | Yes — log commits | Yes — monotonic snapshot rows; begin/end validity on catalog rows |
| Time travel | Yes | Yes — AT (VERSION => n) / timestamp attach |
| Schema evolution | Yes | Yes — field-id remapping; lossless type promotions |
| Deletes / updates | Copy-on-write and merge-on-read modes in the ecosystem | Merge-on-read delete files (Parquet positional; experimental puffin vectors) |
| First-class engine today | Spark ecosystem; broader multi-engine surface | DuckDB first; multi-engine story still early |
| Built-in FGAC (RBAC / RLS / masks) | Mature in Unity Catalog + Delta stacks | Not in the format — catalog GRANT/RLS + object-store paths; real row/column needs a proxy |
| Operational shape | Platform-heavy when you buy the full stack | Three pieces: engine, SQL catalog, object storage (see Field Note 04) |
The governance hole is real
Official DuckLake guidance is explicit: access control is not a feature of the format. You leverage the catalog database and the storage backend. The documented role pattern is superuser / writer / reader on Postgres, plus path-scoped object-store policies under /schema/table/…. Maintainers have said fine-grained access control belongs in an external service — for example something in the Unity Catalog class — not in the open table format itself. That is the correct boundary for a format. It is also the reason DuckLake is not “Delta + UC” under another name.
Catalog, storage, query edge
If you only secure one of the three, you do not have multi-tenant governance. You have a demo.
Layer A
Postgres GRANT + RLS
Controls who can commit and which catalog rows are visible. RLS on ducklake_table can hide tables by region or tenant. It does not filter rows inside Parquet. Tags in ducklake_tag are labels, not policies.
Layer B
Object-store path IAM
Restricts Get/Put/Delete to prefixes. Without this, a leaked HMAC bypasses every catalog policy. Path policies fail for adopted files that ignore the default layout.
Layer C
Query proxy / Flight edge
Rewrites SQL for row filters and column masks; holds service credentials so clients never touch the bucket. This is where real ABAC lands. Ecosystem options range from table-level guards to full RBAC graphs with experimental row/column rewrite.
Anti-pattern
Catalog RLS alone
Production writeups (including teams running DuckLake on managed Postgres + object storage) treat catalog RLS as metadata security, not data security. Design the dual surface on day one.
For a concrete catalog-RLS pattern: create Postgres roles first, attach DuckLake so the ducklake_* tables exist, then enable RLS. Use a permissive baseline policy before restrictive ones. Flush inlined rows to Parquet before you demo storage. Wrap CREATE TABLE + first insert in a transaction so a writer without PUT rights fails cleanly instead of leaving empty catalog ghosts. That is schema/table visibility — not attribute-based row security.
Pick by the job, not the slogan
The useful decision rule is boring. Choose Delta (or Iceberg) when multi-engine federation and mature enterprise governance are non-negotiable. Choose DuckLake when the primary engine is DuckDB, the team can run Postgres, the cost and operational surface matter more than a decade of Spark ecosystem glue, and you are willing to design access control deliberately.
| Situation | Lean Delta / Iceberg | Lean DuckLake |
|---|---|---|
| Spark + Trino + Flink against one table set | Default | Too early |
| DuckDB-first analytics, single-node or small fleet | Possible, heavier | Default |
| Need Unity-style RBAC, masks, lineage tomorrow | Default | Only with a proxy you own |
| Want catalog inspectable as SQL, minimal control plane | Fight the design | Default |
| European mid-market cost pressure (see Field Note 03 / 04) | Often overbuilt | Strong fit |
| Petabyte scans that truly need a large cluster | Safer | Wrong compute model |
Where the claim should make you stop
Four limits. None of them kill DuckLake. All of them kill the lazy version of the title.
Limit 01
Governance maturity
Table-level patterns and Postgres RLS exist. Full RBAC + row filters + column masks require a query edge. The ecosystem is active (catalog guards, FlightSQL gateways, custom proxies). It is not plug-and-play Unity.
Limit 02
Multi-engine breadth
Delta’s power is years of multi-engine production. DuckLake’s first-class path is DuckDB. If your architecture is “any engine, same table,” stay with the formats built for that fight.
Limit 03
Scale shape
DuckDB is a single-machine engine. A workload that genuinely needs a large distributed cluster is not a DuckLake-vs-Delta question — it is a compute question. Do not use a format choice to deny physics.
Limit 04
Operational youth
DuckLake v1.0 is young relative to Delta’s battle history. Catalog size, compaction discipline, and edge-case correctness still need measurement on your data, not on a conference slide.
New Delta means a new catalog, not a new religion
Delta Lake won because it made open ACID tables on object storage a boring, shippable contract. DuckLake is the same contract with a different catalog philosophy: the bookkeeping is a SQL database, the data is still Parquet, the format refuses to become a governance product. That is why the title works for architecture. It is also why the title fails if you skip access control design.
If you are evaluating DuckLake after Field Note 04’s cost argument, do the next experiment on purpose. Pick one table set. Attach with Postgres. Enforce superuser / writer / reader. Add path-scoped storage credentials. Decide whether catalog RLS is enough for your threat model. If you need row filters or column masks, plan the query edge before the first multi-tenant user lands — not after the first audit finding.
The deeper reframe is not “DuckLake versus Delta.” It is “which layer owns which job?” Open table formats should own ACID data on open files. Databases should own transactional metadata when that is the cheaper truth store. Governance products should own identity, grants, masks, and audit. The industry spent a decade bundling those jobs into one invoice. DuckLake is interesting because it unbundles them again. That is what “new Delta” should mean — if it means anything useful at all.
— end —