Your Databricks Lakehouse Is a Postgres Table and a Parquet Folder
A working note for anyone who signs a cloud bill. The lakehouse architecture most companies pay €200,000 a year for is two things most of them already operate. The performance and the cost story are better when you stop pretending otherwise.
A lakehouse is, in the end, a very small idea. It is a folder of Parquet files on object storage, plus a way of remembering which file is the current truth, which files used to be the truth, and which rows have been deleted, inserted, or updated since then. The first part — the Parquet folder — most engineering teams already know how to run. The second part — the memory of the truth — is a SQL database. That is the whole architecture. The reason it costs what it costs is not the architecture. It is what the industry decided to charge for telling you that the architecture is hard.
The lakehouse is two things. A folder of Parquet, and a SQL table that knows which folder is current. The €200,000-a-year product is the part that hides that fact from you.
— closing observationWhat a lakehouse actually is, in two boxes
Strip the marketing. A lakehouse is a data layer that gives you two things at the same time: cheap storage for huge files, and the SQL features of a database (transactions, schema, time travel, updates). The clever move is to split those two jobs. The data lives as Parquet files in object storage — S3, Hetzner Storage Box, MinIO, anything. The bookkeeping for the data — which file is current, which rows changed, what the schema is — lives in a catalog. The interesting question is what the catalog is made of.
Parquet and a SQL table
Every lakehouse in production, including the €200,000-a-year kind, is these two pieces. The disagreement is over what the SQL table is made of. One answer is a Postgres database. The other is a managed control plane that bills per query.
The data
A folder of Parquet files
Open format. Readable by anything that can read a file. Cheap to store. Slow to scan without a good engine. The boring half of the architecture.
The memory
A SQL table that knows the truth
Snapshots, schemas, row counts, which file is current, which rows are deleted. Stored as rows in a real database — or, in the traditional answer, as JSON and Avro files scattered across object storage.
Why a Databricks lakehouse costs what it costs
Apache Iceberg, the open format most teams pick, stores its catalog as files on the same object storage as the data. Snapshots are JSON manifests. Schemas are Avro. The design was made in 2017 to avoid running a database — a reasonable choice in the cloud era of 2017, when managed Postgres was expensive and Hive Metastore was the only option. Ten years later, the cloud has Postgres as a service for €25 a month, and the design has not changed. The cost of pretending it has is significant.
Layer 01
The compute layer
A cluster of rented machines running Spark or Photon. The same machinery described in Field Note 03. Always on, billed by the hour, tuned by a small team of people who have a very specific job title.
Layer 02
The catalog service
Unity Catalog, AWS Glue, or a self-hosted REST catalog. A second fleet of machines whose only job is to remember which file is current. Reads and writes go through it on every single operation.
Layer 03
The metadata files
Thousands of small JSON and Avro files in S3. Every commit is a new file. Every query has to walk a tree of them to plan. The cost shows up in two places: storage of the small files, and the wall-clock time of every query.
Layer 04
The vendor fee
DBU fees on top of the VM cost. License fees for the catalog. A separate bill for the workspace. The pattern is: charge for the machine, then charge for looking at the machine.
DuckLake: same shape, smaller stack
DuckLake, released as a v1.0 by the DuckDB team in April 2026, makes the same architectural bet — Parquet on object storage, plus a catalog that knows the truth — but stores the catalog in a real SQL database. PostgreSQL is the default. DuckDB itself or SQLite work as well. The catalog tables are normal tables. Snapshots are rows. Schemas are rows. The whole bookkeeping layer is a schema you can read with a SELECT statement. That single decision removes the catalog service, the REST API, and the tree of small JSON files. The control plane was always a SQL table pretending to be something else. DuckLake stops pretending.
| Dimension | Traditional lakehouse (Databricks on Azure West Europe) | DuckLake on a Hetzner box + Postgres |
|---|---|---|
| Compute layer | Always-on Spark cluster, 4–16 nodes, 8 vCPU / 32 GB each. €1,400–€5,600/month before DBU fees. | One Hetzner AX162 or SX135, 12–64 cores, 128–256 GB RAM. €239–€470/month. DuckDB runs the queries. |
| Catalog service | Unity Catalog or a self-hosted REST catalog. A second fleet of machines, billed separately. €300–€1,200/month. | None. The catalog is a schema in your Postgres. The cost is included in the €25/month managed Postgres. |
| Metadata storage | Thousands of small JSON and Avro files in S3. Costs add up: storage of the small files, plus GET requests on every query. | Rows in a Postgres table. No small files. No GET requests. A single SELECT replaces the tree walk. |
| Vendor / software fees | DBU fees on top of the VM cost. For an always-on cluster, these typically add as much as the VMs themselves. | Zero. DuckLake is an open format. DuckDB and Postgres are open source. The DuckLake extension is open source. |
| Total, per month | €3,000–€8,000 A working number for a mid-market team running an always-on lakehouse, including DBU and catalog fees. | €280–€510 Hetzner box, managed Postgres, S3-compatible storage. Same Parquet data, same queryable result, much smaller bill. |
| Total, per year | €36,000–€96,000 — before counting the engineer-hours spent keeping the cluster and the catalog happy. | €3,360–€6,120. The same engineer can run the whole stack from a single config file. |
| Where the data lives | Azure West Europe. Subject to US cloud act, to egress fees, to the provider's interpretation of cross-border transfer rules. | Hetzner Storage Box in Germany, or Scaleway in Paris, or OVHcloud in Roubaix. GDPR-native. No US jurisdiction. Egress free or near-free. |
| Hidden cost | JVM tuning, executor crashes, shuffle debugging, Unity Catalog permissions, delta-sync jobs, vendor lock-in to the workspace. | One Python process, one Postgres database, one folder of Parquet. The format is open. The data is yours. The exit takes an afternoon. |
Why the smaller stack is also the faster one
The performance argument is the part that surprises people. A single €239/month Hetzner box running DuckDB, on Parquet files in Hetzner Storage Box, with a Postgres catalog across the room, is faster than a cluster for the workloads most teams actually run. Not "comparable." Faster. The reason is the same reason as in Field Note 03: the cluster spends most of its time coordinating. DuckLake removes a second kind of coordination — the catalog lookup — by storing the catalog in a SQL database the engine can talk to directly.
The workload numbers, not the marketing numbers
Three independent measurements. The honest reading is that DuckLake wins decisively on writes and on small, frequent updates, ties on reads, and is moving fast on the remaining gaps.
Measurement 01
Tunguz · 1 PB, 100M files
A working benchmark on a production-shape dataset: sub-second query planning across a petabyte. The bottleneck in the traditional stack is the catalog walk, not the data scan.
Measurement 02
Radboud University · 2026 thesis
Kafka to lakehouse ingestion. DuckLake: 9.9 ms median write time. Iceberg via DuckDB: 124.3 ms. A ~17× throughput advantage on individual events, with a real academic methodology.
Measurement 03
Inlining on streaming writes
Small inserts go into Postgres instead of triggering a new Parquet file. The official measurement: 926× faster on streaming-micro-batch queries vs. Iceberg. Independent measurement: 7.5% on individual events, which is still meaningful.
Measurement 04
Read parity, with one caveat
On byte-identical Parquet, reads are at parity. The caveat: DuckLake's optimizer can produce bad plans when statistics are stale. A known open issue. Fixable, not fixed.
Why the open format + a European box is a different deal
The cost story changes shape when the cloud provider is European. Hetzner, OVHcloud, Scaleway, and IONOS are 3–8× cheaper than the US hyperscalers for the same raw compute, with no egress fees to speak of, with GDPR-native data residency, and with infrastructure that is, on average, twenty minutes from a Dutch office. The reason US providers charge what they charge is not the compute. It is the bundle — the catalog, the control plane, the workspace, the per-job fees. Once you stop needing the bundle, the raw compute comparison becomes the whole comparison. The bill collapses.
| Provider | What it offers for the DuckLake stack | Indicative monthly cost | Jurisdiction |
|---|---|---|---|
| Hetzner (Falkenstein, Germany) | AX162 / SX135 dedicated server, Storage Box (€3.81/TB/month), managed Postgres available via partners. | €239–€470 + €40/TB | Germany · GDPR-native · no US cloud act |
| OVHcloud (Roubaix / Gravelines, France) | SoYouStart dedicated servers, Object Storage (S3-compatible, egress-free within EU), managed Postgres. | €220–€450 + €20/TB | France · SecNumCloud certified |
| Scaleway (Paris, France) | Dedicated servers, Object Storage (€0.01/GB/month, no egress fees within EU), managed Postgres. | €280–€500 + €10/TB | France · GDPR-native · EU AI Act ready |
| IONOS (Berlin, Germany) | Dedicated servers, S3-compatible storage, managed databases. Smaller scale, competitive on price. | €200–€400 + €30/TB | Germany · GDPR-native |
| Azure West Europe (for context) | Same workload, bundled with Unity Catalog, Photon, and the DBU pricing model. | €3,000–€8,000 | Netherlands region · US parent (Cloud Act applies) |
| AWS eu-central-1 (for context) | Same workload, bundled with Glue catalog, S3, and the EMR / Athena pricing model. | €2,800–€7,500 | Frankfurt region · US parent (Cloud Act applies) |
Three things fall out of the European option that the US comparison does not capture. The first is jurisdiction: data on a Hetzner box in Falkenstein is governed by German and EU law, not by the US Cloud Act. For HR data, for patient data, for any data that crosses a regulatory border, this is not a theoretical concern. The second is egress: Hetzner, OVHcloud, and Scaleway either have no egress fees or charge a fraction of AWS or Azure. The third is latency: the round-trip from Rotterdam to Frankfurt is under 10 ms. A Hetzner box in Germany is faster to a Dutch office than the AWS Frankfurt region is to most of the same office's other cloud calls.
What disappears when you stop renting the bundle
The cost reduction is not the whole argument. The simplification is. A traditional lakehouse on Azure or AWS is, conservatively, six services that need to be configured, secured, versioned, monitored, and explained to an auditor. A DuckLake stack on a European box is three. The difference between six and three is not a 50% reduction in operational complexity. It is the difference between a platform team and a single data engineer.
Six services, or three
The traditional stack has six moving parts, each of which is a vendor relationship, a credential, an SLA, and a thing that can go wrong at 3am. The DuckLake stack has three. The three are open source.
The traditional stack
Six things to operate
Compute (Databricks / EMR). Catalog (Unity / Glue / REST). Storage (S3 / ADLS). Workspace (Databricks UI / EMR Studio). Permissions (Unity Catalog IAM). Metadata store (Glue Data Catalog / HMS).
The DuckLake stack
Three things to operate
Compute (DuckDB on a Hetzner box). Catalog (Postgres on the same network). Storage (Hetzner Storage Box, or any S3-compatible bucket). That is the whole list. The data files are Parquet. The catalog is a schema.
The other thing that disappears is the per-job fee. Databricks charges DBU, AWS charges per Athena query, Snowflake charges per credit. The pattern is the same: charge for the data, then charge for looking at the data. On a DuckLake stack, there is no per-job fee. The compute is a machine you already rented. The query is free. The data team is no longer incentivised to think about whether the analyst is allowed to ask a question. The answer is always yes, because the question is free.
Where DuckLake is not the answer
The argument has a ceiling. DuckLake is a year old, not battle-tested at the scales Databricks has run, and missing some of the integrations the traditional stack has built over a decade. Pretending otherwise is a sales pitch. Four honest limits.
Limit 01
Multi-engine federation
If your queries need to run on Spark, Trino, Flink, and Snowflake against the same tables, Iceberg is the safer answer today. DuckLake's Polars and Trino integrations are early. DuckDB is the first-class engine.
Limit 02
Petabyte single-machine workloads
DuckDB is a single-machine engine. A petabyte scan that genuinely needs to spread across a hundred cores needs a cluster. The Tunguz benchmark is one data point on one workload. Most Dutch mid-market teams are not at this scale.
Limit 03
Correctness on edge cases
Two open issues as of June 2026: a cross-store delete conflict in the v1.0 extension, and a Postgres 1600-column ceiling for the inlined-data table. Both are documented. Neither is fixed in the current release.
Limit 04
Operational maturity of the catalog
A DuckLake catalog at 1 PB is roughly a billion rows in Postgres. Operationally manageable, but a real cost Iceberg avoids. The math has to be done for your specific scale, not assumed.
The lakehouse is two things. Run them as two things.
The lakehouse is a Parquet folder and a SQL table. The traditional stack charges you to pretend that is hard. DuckLake does not pretend. The result is a stack that is 80–90% cheaper to run, faster on the workloads most teams actually have, GDPR-native when you put it on a European box, and small enough that a single data engineer can operate it end to end. The bill, the team, the regulator, and the exit all get simpler at the same time.
If you are a Dutch data lead, the path is short. Pick one of your smallest lakehouse workloads. Check the actual Parquet size, the actual query time, the actual annual cost of the cluster plus the catalog plus the workspace. Spin up a Hetzner AX162 for €239 a month, install Postgres and DuckDB with the DuckLake extension, point it at a Hetzner Storage Box, and run the same transformation. The measurement will make the decision obvious. The thing you are paying for is the part that was always two things.
The deeper shift is in how you frame the question. A lakehouse is not a product. It is an architecture. The architecture is a Parquet folder, a SQL table, and an engine to ask questions of both. Three open-source things. The €200,000-a-year version of this is the same three things, with a vendor in the middle charging for the privilege of pretending the three things are hard. They are not hard. The bill you are signing this quarter has not caught up.
— end —