Vol. 01 · No. 07
— Field Notes —
July 2026
A Field Note on Open Table Formats

DuckLake is the new Delta Lake

I mean this as a working claim, so the limits are in here too. Delta Lake made open ACID lakehouses on object storage mainstream. DuckLake does the same job with a SQL catalog instead of a log of small metadata files, and it leaves governance to you. It follows on from Field Note 04.

Every few years the industry renames the same stack and acts like the new name is a product, and “lakehouse” was one of those. Delta Lake was different, because it made a concrete contract with ACID tables on Parquet, time travel and schema evolution, and it let engines share the same files. I think DuckLake, from the DuckDB team, is the next concrete contract worth taking seriously. It stores the catalog as ordinary SQL tables in Postgres (or DuckDB, or SQLite), keeps the data as Parquet and won’t ship a Unity-style control plane as part of the format. That makes it either the new Delta or a half-finished one, and honestly both can be true at once.

Delta Lake made the open lakehouse real. DuckLake does the same job with a SQL table as the catalog, and you still have to design the governance yourself.

— working claim
01 · The claim

What “new Delta” is allowed to mean

If “new Delta” means “replace Databricks tomorrow,” then the claim is false. If it means “the open table format that will absorb the next wave of mid-market lakehouses the way Delta absorbed the last,” then I think it’s a serious design argument. The rest of this note makes that argument, with the limits left in.

Three things the claim is not

No rebrand, no vendor and no governance product

DuckLake is an open table format specification plus a DuckDB extension. The catalog is a SQL database you already know how to run, and the data is Parquet. The format doesn’t ship RBAC, row filters or column masks as first-class features.

Allowed meaning

Same architectural job as Delta

ACID tables on object storage, snapshots, time travel, schema evolution without rewriting every file and multi-writer coordination through a catalog. That’s what Delta turned into an industry standard for the open lakehouse era.

Disallowed meaning

Same product surface as Delta + UC

Unity Catalog’s RBAC, lineage and multi-engine governance aren’t DuckLake features. The official docs and the maintainers treat fine-grained access control as out of scope for the format, so the gap is part of the design, and I wouldn’t expect it to close with time.

Figure 1 Define the claim before you argue it. “New Delta” is about the open table format job, and I’m not claiming that DuckLake is Unity Catalog, Photon or a Databricks workspace.
02 · What Delta solved

The problem Delta Lake actually fixed

Before Delta, “data lake” often meant a bucket of files with no reliable transactions. Concurrent writers raced each other, readers saw half-written partitions and time travel was basically a folder naming convention. Delta’s answer was a transaction log, with commits as ordered records, the data still in Parquet (and related columnar forms) and engines reading the log to know what’s current. That contract scaled across Spark and later a wider set of engines, and it became the default mental model for an open lakehouse.

Contract 01

ACID on object storage

Commits are atomic and readers don’t see partial writes. Multiple writers coordinate through the log instead of just hoping for the best.

Contract 02

Time travel as a product feature

You can query as of a version, roll back and audit what changed, so the lake stops being a write-only dumping ground.

Contract 03

Schema evolution without panic

Add columns, rename carefully, evolve the table without rewriting history for every small change.

Contract 04

Open files, managed metadata

The data stays in open formats and the metadata protocol is really the product. Governance layers (Unity Catalog and its peers) then sat on top of that protocol for enterprises.

Figure 2 What Delta really contributed was the contract. Any format that wants to be “the new Delta” has to ship those four properties in a form teams can run without paying a platform tax forever.
03 · The catalog bet

DuckLake’s move is to make the catalog a database

DuckLake keeps the Delta-era split between data files and bookkeeping, but it doesn’t use the small-file metadata tree as its main design. Snapshots, schemas, columns, data files, delete files, partition values and stats all live as rows in a catalog database, and v1.0 defines 28 catalog tables. Validity is tracked with begin/end snapshot intervals. Concurrent writers race on a monotonic snapshot id, and the extension retries when there’s no logical conflict. So SQL really is the catalog here, with no REST catalog hiding behind a SQL UI.

Two open formats, two catalog philosophies

Log files vs rows

Both designs put Parquet (or something similar) on object storage. Where they disagree is where the truth lives, and how expensive it is to ask “which files are current?”

Delta / Iceberg style

Metadata as files on the lake

Transaction logs, manifests and metadata trees on object storage. It avoids needing a database, but it pays for that with lots of small GETs and a separate catalog service once you want multi-user governance.

DuckLake style

Metadata as SQL tables

Postgres (the preferred option for multiple users), DuckDB or SQLite for the catalog. One SELECT replaces a tree walk, file pruning uses min/max stats stored as rows, and you can inspect the catalog with ordinary SQL tools.

Figure 3 This is the structural claim behind “new Delta.” It’s the same open-lakehouse job with a different answer to “what is a catalog made of?” Field Note 04 covered what that answer means for cost, and this note covers what it means for the product category.
04 · Side by side

Where DuckLake is Delta’s peer, and where it isn’t

A category claim needs a table to back it up. The comparison below is only about the open table format job, so it leaves out Databricks workspaces, Photon and Spark clusters, which are compute products that often sit on top of Delta. DuckDB sits on top of DuckLake the same way Spark sits on top of Delta. I think mixing up the format and the compute is exactly how the bill stays hard to read.

Dimension Delta Lake (open format + typical enterprise stack) DuckLake (open format + typical mid-market stack)
Data files Parquet (and related columnar) on object storage Parquet on any filesystem DuckDB can reach (local, S3, GCS, Azure, R2, etc.)
Catalog Transaction log + metastore / Unity Catalog / REST catalogs 28 versioned SQL tables in Postgres, DuckDB, or SQLite
ACID / snapshots Yes, log commits Yes, monotonic snapshot rows, with begin/end validity on catalog rows
Time travel Yes Yes, AT (VERSION => n) / timestamp attach
Schema evolution Yes Yes, field-id remapping and lossless type promotions
Deletes / updates Copy-on-write and merge-on-read modes in the ecosystem Merge-on-read delete files (Parquet positional, experimental puffin vectors)
First-class engine today Spark ecosystem, with a broader multi-engine surface DuckDB first, and the multi-engine story is still early
Built-in FGAC (RBAC / RLS / masks) Mature in Unity Catalog + Delta stacks Not in the format, so catalog GRANT/RLS plus object-store paths, and real row/column control needs a proxy
Operational shape Platform-heavy when you buy the full stack Three pieces: engine, SQL catalog, object storage (see Field Note 04)
Figure 4 A peer on the open table format, but not on enterprise governance. That one row is why “DuckLake is the new Delta” is a useful claim for architecture teams and a dangerous one for compliance teams.
05 · Access control

The governance hole is real

The official DuckLake guidance is pretty explicit that access control isn’t a feature of the format. You use the catalog database and the storage backend for it instead. The documented role pattern is superuser / writer / reader on Postgres, plus path-scoped object-store policies under /schema/table/…. The maintainers have said fine-grained access control belongs in an external service, for example something in the Unity Catalog class, and not in the open table format itself. I think that’s the right boundary for a format, and it’s also the reason DuckLake isn’t “Delta + UC” under another name.

Three enforcement layers

Catalog, storage, query edge

If you only secure one of the three, you don’t have multi-tenant governance yet, you just have a demo.

Layer A

Postgres GRANT + RLS

Controls who can commit and which catalog rows are visible. RLS on ducklake_table can hide tables by region or tenant, but it doesn’t filter rows inside Parquet. Tags in ducklake_tag are just labels, they don’t enforce anything.

Layer B

Object-store path IAM

Restricts Get/Put/Delete to prefixes. Without this, a leaked HMAC gets around every catalog policy. Path policies also fail for adopted files that don’t follow the default layout.

Layer C

Query proxy / Flight edge

Rewrites SQL for row filters and column masks, and holds the service credentials so clients never touch the bucket. This is where real ABAC ends up. The options in the ecosystem range from table-level guards to full RBAC graphs with experimental row/column rewrites.

Anti-pattern

Catalog RLS alone

Production writeups (including teams running DuckLake on managed Postgres + object storage) treat catalog RLS as security for the metadata, and something else has to protect the data. Design both sides on day one.

Figure 5 Unity Catalog doesn’t “sit on top of” DuckLake. Both are catalog layers. If you need UC’s governance model, you use UC with Delta, and if you need DuckLake’s operational model, you build the governance yourself at the catalog + storage + proxy boundary.

If you want a concrete catalog-RLS pattern, create the Postgres roles first, attach DuckLake so the ducklake_* tables exist, and then enable RLS. Use a permissive baseline policy before the restrictive ones, and flush inlined rows to Parquet before you demo storage. Wrap CREATE TABLE and the first insert in a transaction, so a writer without PUT rights fails cleanly instead of leaving empty ghosts in the catalog. Keep in mind that this only gives you schema/table visibility, and it’s not attribute-based row security.

06 · When to choose

Pick by the job and ignore the slogan

The useful decision rule is fairly boring. Choose Delta (or Iceberg) when multi-engine federation and mature enterprise governance are non-negotiable. Choose DuckLake when DuckDB is the main engine, the team can run Postgres, cost and operations matter more than a decade of Spark ecosystem glue, and you’re willing to design access control on purpose.

Situation Lean Delta / Iceberg Lean DuckLake
Spark + Trino + Flink against one table set Default Too early
DuckDB-first analytics, single-node or small fleet Possible, heavier Default
Need Unity-style RBAC, masks, lineage tomorrow Default Only with a proxy you own
Want catalog inspectable as SQL, minimal control plane Fight the design Default
European mid-market cost pressure (see Field Note 03 / 04) Often overbuilt Strong fit
Petabyte scans that truly need a large cluster Safer Wrong compute model
Figure 6 Category claims fall apart in the edge cases. DuckLake is the new Delta for teams whose real workload fits DuckDB + SQL catalog + open Parquet. For teams whose real product is multi-engine enterprise governance, it isn’t.
07 · Honest limits

Where the claim should make you stop

There are four limits. None of them kill DuckLake, but all of them kill the lazy version of the title.

Limit 01

Governance maturity

Table-level patterns and Postgres RLS exist, but full RBAC + row filters + column masks need a query edge. The ecosystem is active (catalog guards, FlightSQL gateways, custom proxies), it just isn’t plug-and-play like Unity.

Limit 02

Multi-engine breadth

Delta’s strength is years of multi-engine production, while DuckLake’s first-class path is DuckDB. If your architecture is “any engine, same table,” I’d stay with the formats built for that.

Limit 03

Scale shape

DuckDB is a single-machine engine. If a workload genuinely needs a large distributed cluster, the real question is compute, and picking DuckLake or Delta won’t change the physics.

Limit 04

Operational youth

DuckLake v1.0 is still young compared to Delta’s track record. Catalog size, compaction discipline and edge-case correctness still need to be measured on your own data, and a conference slide won’t tell you.

Figure 7 Honesty is the difference between a Field Note and a launch blog. The title is a category claim, and the limits are the requirements document that should sit right next to it.
Closing observation

New Delta means a new catalog, and no new religion

Delta Lake won because it turned open ACID tables on object storage into a boring contract you could actually ship. DuckLake is the same contract with a different idea of the catalog, where the bookkeeping is a SQL database, the data is still Parquet and the format won’t become a governance product. That’s why the title works for architecture, and it’s also why the title fails if you skip designing access control.

If you’re evaluating DuckLake after the cost argument in Field Note 04, I’d do the next experiment on purpose. Pick one table set, attach it with Postgres, enforce superuser / writer / reader and add path-scoped storage credentials. Then decide whether catalog RLS is enough for your threat model. If you need row filters or column masks, plan the query edge before the first multi-tenant user shows up, instead of after the first audit finding.

I think the more useful question is “which layer owns which job?” Open table formats should own ACID data on open files. Databases should own transactional metadata when they’re the cheaper place to keep the truth, and governance products should own identity, grants, masks and audit. The industry spent a decade bundling those jobs into one invoice, and DuckLake is interesting because it unbundles them again. That’s what “new Delta” should mean, if it means anything useful at all.

— end —

DuckLake is the new Delta Lake · A Field Note Set in Shadows Into Light Two & Cause

← Back to all posts