Vol. 01 · No. 04
— Field Notes —
June 2026
A Field Note on Data Architecture

Your Databricks Lakehouse Is a Postgres Table and a Parquet Folder

A working note for anyone who signs a cloud bill. A mid-market lakehouse can easily cost €36,000–€96,000 a year (my working figure, not a quote), but it’s really two things most teams already run. Both the performance and the cost look better once you stop pretending otherwise.

In the end a lakehouse is a pretty small idea. It’s a folder of Parquet files on object storage, plus a way of remembering which file is the current truth and which files used to be. It also tracks which rows have been deleted, inserted or updated since then. Most engineering teams already know how to run the Parquet folder, and the memory of the truth is just a SQL database. I think what makes it cost what it costs is what the industry decided to charge for telling you the architecture is hard.

A lakehouse is a folder of Parquet and a SQL table that knows which folder is current, and the expensive product is mostly the part that hides that from you.

— closing observation
01 · The setup

What a lakehouse actually is, in two boxes

Without the marketing, a lakehouse is a data layer that gives you two things at once, cheap storage for huge files and the SQL features of a database (transactions, schema, time travel, updates). The clever part is splitting those two jobs. The data lives as Parquet files in object storage, so S3, Hetzner Storage Box, MinIO or anything else. The bookkeeping, meaning which file is current, which rows changed and what the schema is, lives in a catalog. So the interesting question is what the catalog is made of.

The two things a lakehouse actually is

Parquet and a SQL table

Every lakehouse in production, including the very expensive kind, is these two pieces. Where they differ is what the SQL table is made of, and one answer is a Postgres database while the other is a managed control plane that bills per query.

The data

A folder of Parquet files

An open format that anything able to read a file can read. It’s cheap to store but slow to scan without a good engine, so it’s the boring half of the architecture.

The memory

A SQL table that knows the truth

Snapshots, schemas, row counts, which file is current and which rows are deleted. These are stored as rows in a real database, or in the traditional answer as JSON and Avro files scattered across object storage.

Figure 1 The whole architecture, in two boxes. Everything else, like the control plane, the catalog service, Unity Catalog or the REST catalog, is a way of either storing or hiding the SQL table. DuckLake’s argument is that the hiding is the expensive part.
02 · The traditional answer

Why a Databricks lakehouse costs what it costs

Apache Iceberg, the open format most teams pick, stores its catalog as files on the same object storage as the data, with snapshots as JSON manifests and schemas in Avro. The design was made in 2017 to avoid running a database, which was a reasonable choice at the time, when managed Postgres was expensive and Hive Metastore was the only option. Ten years later the cloud has Postgres as a service for €25 a month, and the design still hasn’t changed. Pretending otherwise turns out to be pretty expensive.

Layer 01

The compute layer

A cluster of rented machines running Spark or Photon, the same machinery described in Field Note 03. It’s always on, billed by the hour and tuned by a small team of people with a very specific job title.

Layer 02

The catalog service

Unity Catalog, AWS Glue or a self-hosted REST catalog, so a second fleet of machines whose only job is to remember which file is current. Reads and writes go through it on every single operation.

Layer 03

The metadata files

Thousands of small JSON and Avro files in S3. Every commit is a new file, and every query has to walk a tree of them to plan. The cost shows up in two places, the storage of all those small files and the wall-clock time of every query.

Layer 04

The vendor fee

DBU fees on top of the VM cost, license fees for the catalog and a separate bill for the workspace. The pattern is to charge for the machine, and then charge again for looking at the machine.

Figure 2 What you’re actually paying for. A lakehouse on the traditional stack is four moving parts, and each one is a team and a budget line. The cost adds up because the parts talk to each other on every operation.
03 · The other answer

DuckLake has the same shape with a smaller stack

DuckLake was released as v1.0 by the DuckDB team in April 2026. It makes the same architectural bet, Parquet on object storage plus a catalog that knows the truth, but it stores that catalog in a real SQL database. PostgreSQL is the default, and DuckDB itself or SQLite work as well. The catalog tables are normal tables, so snapshots and schemas are just rows, and the whole bookkeeping layer is a schema you can read with a SELECT statement. That one decision removes the catalog service, the REST API and the tree of small JSON files. I’d say the control plane was always a SQL table pretending to be something else, and DuckLake simply stops pretending.

Dimension Traditional lakehouse (Databricks on Azure West Europe) DuckLake on a Hetzner box + Postgres
Compute layer Always-on Spark cluster, 4–16 nodes, 8 vCPU / 32 GB each. €1,400–€5,600/month before DBU fees. One Hetzner AX162 or SX135, 12–64 cores, 128–256 GB RAM. €239–€470/month. DuckDB runs the queries.
Catalog service Unity Catalog or a self-hosted REST catalog. A second fleet of machines, billed separately. €300–€1,200/month. None. The catalog is a schema in your Postgres. The cost is included in the €25/month managed Postgres.
Metadata storage Thousands of small JSON and Avro files in S3, and the costs add up with the storage of the small files plus GET requests on every query. Rows in a Postgres table, so no small files and no GET requests. A single SELECT replaces the tree walk.
Vendor / software fees DBU fees on top of the VM cost. For an always-on cluster, these typically add as much as the VMs themselves. Zero. DuckLake is an open format, and DuckDB, Postgres and the DuckLake extension are all open source.
Total, per month €3,000–€8,000 A working number for a mid-market team running an always-on lakehouse, including DBU and catalog fees. €280–€510 Hetzner box, managed Postgres and S3-compatible storage. The same Parquet data and the same queryable result, with a much smaller bill.
Total, per year €36,000–€96,000, before counting the engineer-hours spent keeping the cluster and the catalog happy. €3,360–€6,120. The same engineer can run the whole stack from a single config file.
Where the data lives Azure West Europe, subject to the US Cloud Act, to egress fees and to the provider’s interpretation of cross-border transfer rules. Hetzner Storage Box in Germany, Scaleway in Paris or OVHcloud in Roubaix. GDPR-native, no US jurisdiction, and egress is free or close to it.
Hidden cost JVM tuning, executor crashes, shuffle debugging, Unity Catalog permissions, delta-sync jobs, vendor lock-in to the workspace. One Python process, one Postgres database and one folder of Parquet. The format is open, the data is yours and leaving takes an afternoon.
Figure 3 Same architecture, different bill. The data and the queries are cheap. The expensive part is the control plane, meaning the catalog service, the metadata files and the per-job fees. DuckLake removes the parts the traditional stack charges for, because those parts were a workaround for not using a database.
04 · The performance story

Why the smaller stack is also the faster one

The performance argument is the part that surprises people. A single €239/month Hetzner box running DuckDB, on Parquet files in Hetzner Storage Box with a Postgres catalog across the room, is actually faster than a cluster for the workloads most teams run. The reason is the same as in Field Note 03, which is that the cluster spends most of its time coordinating. DuckLake removes a second kind of coordination, the catalog lookup, by storing the catalog in a SQL database the engine can talk to directly.

What the measurement says

What the workload numbers say

There are three independent measurements. My honest reading is that DuckLake clearly wins on writes and on small, frequent updates, ties on reads, and is closing the remaining gaps fast.

Measurement 01

Tunguz · 1 PB, 100M files

A working benchmark on a production-shaped dataset showed sub-second query planning across a petabyte. In the traditional stack the bottleneck is the catalog walk, and the data scan isn’t the problem.

Measurement 02

Radboud University · 2026 thesis

Kafka to lakehouse ingestion, with a 9.9 ms median write time for DuckLake against 124.3 ms for Iceberg via DuckDB. That’s a ~17× throughput advantage on individual events, measured with a real academic methodology.

Measurement 03

Inlining on streaming writes

Small inserts go into Postgres instead of triggering a new Parquet file. The official measurement says 926× faster on streaming micro-batch queries vs. Iceberg, and an independent measurement found 7.5% on individual events, which is still meaningful.

Measurement 04

Read parity, with one caveat

On byte-identical Parquet, reads are at parity. The caveat is that DuckLake’s optimizer can produce bad plans when statistics are stale, which is a known open issue that can be fixed but hasn’t been yet.

Figure 4 Read the workload before the headline. DuckLake isn’t faster at everything, but it is faster at the workloads most Dutch mid-market teams run, like nightly jobs, small updates, frequent commits and large query plans. My honest read of the data is to use DuckLake for the lakehouse and Iceberg for multi-engine federation.
05 · The European angle

Why the open format + a European box is a different deal

The cost story changes when the cloud provider is European. Hetzner, OVHcloud, Scaleway and IONOS are 3–8× cheaper than the US hyperscalers for the same raw compute. They also come with hardly any egress fees, GDPR-native data residency and infrastructure that’s on average twenty minutes from a Dutch office. What US providers really charge for is the bundle around the compute, meaning the catalog, the control plane, the workspace and the per-job fees. Once you stop needing the bundle, the raw compute comparison is the whole comparison, and the bill drops a lot.

Provider What it offers for the DuckLake stack Indicative monthly cost Jurisdiction
Hetzner (Falkenstein, Germany) AX162 / SX135 dedicated server, Storage Box (€3.81/TB/month), managed Postgres available via partners. €239–€470 + €40/TB Germany · GDPR-native · no US cloud act
OVHcloud (Roubaix / Gravelines, France) SoYouStart dedicated servers, Object Storage (S3-compatible, egress-free within EU), managed Postgres. €220–€450 + €20/TB France · SecNumCloud certified
Scaleway (Paris, France) Dedicated servers, Object Storage (€0.01/GB/month, no egress fees within EU), managed Postgres. €280–€500 + €10/TB France · GDPR-native · EU AI Act ready
IONOS (Berlin, Germany) Dedicated servers, S3-compatible storage, managed databases. Smaller scale, competitive on price. €200–€400 + €30/TB Germany · GDPR-native
Azure West Europe (for context) Same workload, bundled with Unity Catalog, Photon, and the DBU pricing model. €3,000–€8,000 Netherlands region · US parent (Cloud Act applies)
AWS eu-central-1 (for context) Same workload, bundled with Glue catalog, S3, and the EMR / Athena pricing model. €2,800–€7,500 Frankfurt region · US parent (Cloud Act applies)
Figure 5 European infrastructure, indicative pricing, June 2026. These are working figures from the public configurators, so they aren’t quotes. What matters is the order of magnitude, since a complete DuckLake stack on European infrastructure costs less in a month than the management fee on a US hyperscaler.

There are three things the European option gives you that the US comparison doesn’t capture. The first is jurisdiction, because data on a Hetzner box in Falkenstein is governed by German and EU law instead of the US Cloud Act. For HR data, patient data or any data that crosses a regulatory border, that’s a real concern and not a theoretical one. The second is egress, since Hetzner, OVHcloud and Scaleway either have no egress fees or charge a fraction of what AWS or Azure do. The third is latency, with the round-trip from Rotterdam to Frankfurt under 10 ms. A Hetzner box in Germany is faster to a Dutch office than the AWS Frankfurt region is to most of the same office’s other cloud calls.

06 · The simplified stack

What disappears when you stop renting the bundle

The cost reduction isn’t the whole argument, and I think the simplification matters even more. A traditional lakehouse on Azure or AWS is, conservatively, six services that need to be configured, secured, versioned, monitored and explained to an auditor, while a DuckLake stack on a European box is three. Going from six to three sounds like a 50% cut in operational complexity, but in practice it’s the difference between needing a platform team and needing a single data engineer.

The stack comparison

Six services, or three

The traditional stack has six moving parts, and each one is a vendor relationship, a credential, an SLA and a thing that can go wrong at 3am. The DuckLake stack has three, and all three are open source.

The traditional stack

Six things to operate

Compute (Databricks / EMR). Catalog (Unity / Glue / REST). Storage (S3 / ADLS). Workspace (Databricks UI / EMR Studio). Permissions (Unity Catalog IAM). Metadata store (Glue Data Catalog / HMS).

The DuckLake stack

Three things to operate

Compute (DuckDB on a Hetzner box), catalog (Postgres on the same network) and storage (Hetzner Storage Box, or any S3-compatible bucket). That’s the whole list, with the data files in Parquet and the catalog as a schema.

Figure 6 The simplification is built into the design. Six services means six credentials, six SLAs, six escalation paths and six lines in the SOC 2 report. Three services means three of each, and a stack the data team can actually reason about end to end.

The other thing that disappears is the per-job fee. Databricks charges DBU, AWS charges per Athena query and Snowflake charges per credit, and the pattern is the same each time, charge for the data and then charge again for looking at it. On a DuckLake stack there’s no per-job fee, because the compute is a machine you already rented and the query is free. So the data team no longer has a reason to think about whether the analyst is allowed to ask a question, and the answer is always yes.

07 · The honest limits

Where DuckLake is not the answer

The argument has a ceiling. DuckLake is a year old, it hasn’t been battle-tested at the scales Databricks has run, and it’s missing some of the integrations the traditional stack has built over a decade. Pretending otherwise would be a sales pitch, so here are four honest limits.

Limit 01

Multi-engine federation

If your queries need to run on Spark, Trino, Flink and Snowflake against the same tables, Iceberg is the safer answer today. DuckLake’s Polars and Trino integrations are still early, and DuckDB is the first-class engine.

Limit 02

Petabyte single-machine workloads

DuckDB is a single-machine engine, so a petabyte scan that genuinely needs to spread across a hundred cores needs a cluster. The Tunguz benchmark is one data point on one workload, and most Dutch mid-market teams aren’t at this scale.

Limit 03

Correctness on edge cases

As of June 2026 there are two open issues, a cross-store delete conflict in the v1.0 extension and a Postgres 1600-column ceiling for the inlined-data table. Both are documented, and neither is fixed in the current release.

Limit 04

Operational maturity of the catalog

A DuckLake catalog at 1 PB is roughly a billion rows in Postgres. That’s manageable to operate, but it’s a real cost that Iceberg avoids, so do the math for your own scale instead of assuming.

Figure 7 Four honest limits. None of them are reasons to avoid DuckLake. They’re reasons to know what you’re picking, and to pick it for the workload that actually fits instead of the one that fits the marketing.
Closing observation

A lakehouse is two things, so run it as two things

A lakehouse is a Parquet folder and a SQL table, and the traditional stack charges you to pretend that’s hard. DuckLake doesn’t pretend. What you get is a stack that’s 80–90% cheaper, faster on most teams’ actual workloads, GDPR-native on a European box and small enough for one data engineer to run end to end. The bill, the team, the regulator and the exit all get simpler at the same time.

If you’re a Dutch data lead, the path is pretty short. Pick one of your smallest lakehouse workloads and check the actual Parquet size, the actual query time and the actual annual cost of the cluster plus the catalog plus the workspace. Then spin up a Hetzner AX162 for €239 a month, install Postgres and DuckDB with the DuckLake extension, point it at a Hetzner Storage Box and run the same transformation. I think the measurement will make the decision for you.

I think the real shift is in how you frame the question. A lakehouse is really an architecture, made of a Parquet folder, a SQL table and an engine to ask questions of both, all three of them open source. The expensive version is the same three things, with a vendor in the middle charging for the privilege of pretending they’re hard. They really aren’t, but the bill you’re signing this quarter hasn’t caught up yet.

— end —

Your Databricks Lakehouse Is a Postgres Table and a Parquet Folder · A Field Note Set in Shadows Into Light Two & Cause

← Back to all posts