DuckLake is the new Delta Lake
I mean this as a working claim, so the limits are in here too. Delta Lake made open ACID lakehouses on object storage mainstream. DuckLake does the same job with a SQL catalog instead of a log of small metadata files, and it leaves governance to you. It follows on from Field Note 04.
Every few years the industry renames the same stack and acts like the new name is a product, and “lakehouse” was one of those. Delta Lake was different, because it made a concrete contract with ACID tables on Parquet, time travel and schema evolution, and it let engines share the same files. I think DuckLake, from the DuckDB team, is the next concrete contract worth taking seriously. It stores the catalog as ordinary SQL tables in Postgres (or DuckDB, or SQLite), keeps the data as Parquet and won’t ship a Unity-style control plane as part of the format. That makes it either the new Delta or a half-finished one, and honestly both can be true at once.
Delta Lake made the open lakehouse real. DuckLake does the same job with a SQL table as the catalog, and you still have to design the governance yourself.
— working claimWhat “new Delta” is allowed to mean
If “new Delta” means “replace Databricks tomorrow,” then the claim is false. If it means “the open table format that will absorb the next wave of mid-market lakehouses the way Delta absorbed the last,” then I think it’s a serious design argument. The rest of this note makes that argument, with the limits left in.
No rebrand, no vendor and no governance product
DuckLake is an open table format specification plus a DuckDB extension. The catalog is a SQL database you already know how to run, and the data is Parquet. The format doesn’t ship RBAC, row filters or column masks as first-class features.
Allowed meaning
Same architectural job as Delta
ACID tables on object storage, snapshots, time travel, schema evolution without rewriting every file and multi-writer coordination through a catalog. That’s what Delta turned into an industry standard for the open lakehouse era.
Disallowed meaning
Same product surface as Delta + UC
Unity Catalog’s RBAC, lineage and multi-engine governance aren’t DuckLake features. The official docs and the maintainers treat fine-grained access control as out of scope for the format, so the gap is part of the design, and I wouldn’t expect it to close with time.
The problem Delta Lake actually fixed
Before Delta, “data lake” often meant a bucket of files with no reliable transactions. Concurrent writers raced each other, readers saw half-written partitions and time travel was basically a folder naming convention. Delta’s answer was a transaction log, with commits as ordered records, the data still in Parquet (and related columnar forms) and engines reading the log to know what’s current. That contract scaled across Spark and later a wider set of engines, and it became the default mental model for an open lakehouse.
Contract 01
ACID on object storage
Commits are atomic and readers don’t see partial writes. Multiple writers coordinate through the log instead of just hoping for the best.
Contract 02
Time travel as a product feature
You can query as of a version, roll back and audit what changed, so the lake stops being a write-only dumping ground.
Contract 03
Schema evolution without panic
Add columns, rename carefully, evolve the table without rewriting history for every small change.
Contract 04
Open files, managed metadata
The data stays in open formats and the metadata protocol is really the product. Governance layers (Unity Catalog and its peers) then sat on top of that protocol for enterprises.
DuckLake’s move is to make the catalog a database
DuckLake keeps the Delta-era split between data files and bookkeeping, but it doesn’t use the small-file metadata tree as its main design. Snapshots, schemas, columns, data files, delete files, partition values and stats all live as rows in a catalog database, and v1.0 defines 28 catalog tables. Validity is tracked with begin/end snapshot intervals. Concurrent writers race on a monotonic snapshot id, and the extension retries when there’s no logical conflict. So SQL really is the catalog here, with no REST catalog hiding behind a SQL UI.
Log files vs rows
Both designs put Parquet (or something similar) on object storage. Where they disagree is where the truth lives, and how expensive it is to ask “which files are current?”
Delta / Iceberg style
Metadata as files on the lake
Transaction logs, manifests and metadata trees on object storage. It avoids needing a database, but it pays for that with lots of small GETs and a separate catalog service once you want multi-user governance.
DuckLake style
Metadata as SQL tables
Postgres (the preferred option for multiple users), DuckDB or SQLite for the catalog. One SELECT replaces a tree walk, file pruning uses min/max stats stored as rows, and you can inspect the catalog with ordinary SQL tools.
Where DuckLake is Delta’s peer, and where it isn’t
A category claim needs a table to back it up. The comparison below is only about the open table format job, so it leaves out Databricks workspaces, Photon and Spark clusters, which are compute products that often sit on top of Delta. DuckDB sits on top of DuckLake the same way Spark sits on top of Delta. I think mixing up the format and the compute is exactly how the bill stays hard to read.
| Dimension | Delta Lake (open format + typical enterprise stack) | DuckLake (open format + typical mid-market stack) |
|---|---|---|
| Data files | Parquet (and related columnar) on object storage | Parquet on any filesystem DuckDB can reach (local, S3, GCS, Azure, R2, etc.) |
| Catalog | Transaction log + metastore / Unity Catalog / REST catalogs | 28 versioned SQL tables in Postgres, DuckDB, or SQLite |
| ACID / snapshots | Yes, log commits | Yes, monotonic snapshot rows, with begin/end validity on catalog rows |
| Time travel | Yes | Yes, AT (VERSION => n) / timestamp attach |
| Schema evolution | Yes | Yes, field-id remapping and lossless type promotions |
| Deletes / updates | Copy-on-write and merge-on-read modes in the ecosystem | Merge-on-read delete files (Parquet positional, experimental puffin vectors) |
| First-class engine today | Spark ecosystem, with a broader multi-engine surface | DuckDB first, and the multi-engine story is still early |
| Built-in FGAC (RBAC / RLS / masks) | Mature in Unity Catalog + Delta stacks | Not in the format, so catalog GRANT/RLS plus object-store paths, and real row/column control needs a proxy |
| Operational shape | Platform-heavy when you buy the full stack | Three pieces: engine, SQL catalog, object storage (see Field Note 04) |
The governance hole is real
The official DuckLake guidance is pretty explicit that access control isn’t a feature of the format. You use the catalog database and the storage backend for it instead. The documented role pattern is superuser / writer / reader on Postgres, plus path-scoped object-store policies under /schema/table/…. The maintainers have said fine-grained access control belongs in an external service, for example something in the Unity Catalog class, and not in the open table format itself. I think that’s the right boundary for a format, and it’s also the reason DuckLake isn’t “Delta + UC” under another name.
Catalog, storage, query edge
If you only secure one of the three, you don’t have multi-tenant governance yet, you just have a demo.
Layer A
Postgres GRANT + RLS
Controls who can commit and which catalog rows are visible. RLS on ducklake_table can hide tables by region or tenant, but it doesn’t filter rows inside Parquet. Tags in ducklake_tag are just labels, they don’t enforce anything.
Layer B
Object-store path IAM
Restricts Get/Put/Delete to prefixes. Without this, a leaked HMAC gets around every catalog policy. Path policies also fail for adopted files that don’t follow the default layout.
Layer C
Query proxy / Flight edge
Rewrites SQL for row filters and column masks, and holds the service credentials so clients never touch the bucket. This is where real ABAC ends up. The options in the ecosystem range from table-level guards to full RBAC graphs with experimental row/column rewrites.
Anti-pattern
Catalog RLS alone
Production writeups (including teams running DuckLake on managed Postgres + object storage) treat catalog RLS as security for the metadata, and something else has to protect the data. Design both sides on day one.
If you want a concrete catalog-RLS pattern, create the Postgres roles first, attach DuckLake so the ducklake_* tables exist, and then enable RLS. Use a permissive baseline policy before the restrictive ones, and flush inlined rows to Parquet before you demo storage. Wrap CREATE TABLE and the first insert in a transaction, so a writer without PUT rights fails cleanly instead of leaving empty ghosts in the catalog. Keep in mind that this only gives you schema/table visibility, and it’s not attribute-based row security.
Pick by the job and ignore the slogan
The useful decision rule is fairly boring. Choose Delta (or Iceberg) when multi-engine federation and mature enterprise governance are non-negotiable. Choose DuckLake when DuckDB is the main engine, the team can run Postgres, cost and operations matter more than a decade of Spark ecosystem glue, and you’re willing to design access control on purpose.
| Situation | Lean Delta / Iceberg | Lean DuckLake |
|---|---|---|
| Spark + Trino + Flink against one table set | Default | Too early |
| DuckDB-first analytics, single-node or small fleet | Possible, heavier | Default |
| Need Unity-style RBAC, masks, lineage tomorrow | Default | Only with a proxy you own |
| Want catalog inspectable as SQL, minimal control plane | Fight the design | Default |
| European mid-market cost pressure (see Field Note 03 / 04) | Often overbuilt | Strong fit |
| Petabyte scans that truly need a large cluster | Safer | Wrong compute model |
Where the claim should make you stop
There are four limits. None of them kill DuckLake, but all of them kill the lazy version of the title.
Limit 01
Governance maturity
Table-level patterns and Postgres RLS exist, but full RBAC + row filters + column masks need a query edge. The ecosystem is active (catalog guards, FlightSQL gateways, custom proxies), it just isn’t plug-and-play like Unity.
Limit 02
Multi-engine breadth
Delta’s strength is years of multi-engine production, while DuckLake’s first-class path is DuckDB. If your architecture is “any engine, same table,” I’d stay with the formats built for that.
Limit 03
Scale shape
DuckDB is a single-machine engine. If a workload genuinely needs a large distributed cluster, the real question is compute, and picking DuckLake or Delta won’t change the physics.
Limit 04
Operational youth
DuckLake v1.0 is still young compared to Delta’s track record. Catalog size, compaction discipline and edge-case correctness still need to be measured on your own data, and a conference slide won’t tell you.
New Delta means a new catalog, and no new religion
Delta Lake won because it turned open ACID tables on object storage into a boring contract you could actually ship. DuckLake is the same contract with a different idea of the catalog, where the bookkeeping is a SQL database, the data is still Parquet and the format won’t become a governance product. That’s why the title works for architecture, and it’s also why the title fails if you skip designing access control.
If you’re evaluating DuckLake after the cost argument in Field Note 04, I’d do the next experiment on purpose. Pick one table set, attach it with Postgres, enforce superuser / writer / reader and add path-scoped storage credentials. Then decide whether catalog RLS is enough for your threat model. If you need row filters or column masks, plan the query edge before the first multi-tenant user shows up, instead of after the first audit finding.
I think the more useful question is “which layer owns which job?” Open table formats should own ACID data on open files. Databases should own transactional metadata when they’re the cheaper place to keep the truth, and governance products should own identity, grants, masks and audit. The industry spent a decade bundling those jobs into one invoice, and DuckLake is interesting because it unbundles them again. That’s what “new Delta” should mean, if it means anything useful at all.
— end —