Vol. 01 · No. 06
— Field Notes —
July 2026
A Field Note on Data Pipelines

Replace Your SQL Pipelines with Polars

Your transformation stack is probably four tools, three languages and a warehouse you can’t really leave. A Polars, Dagster and pytest stack gives you one language, any data source in and out, and data contracts you can enforce in CI before a single row reaches production.

Every data team I know runs a transformation stack that nobody really designed on purpose. There’s Fivetran for ingestion, dbt for SQL models inside the warehouse, Python scripts for the transforms SQL can’t express, and pytest duct-taped to dbt for testing. So the pipeline touches five tools before a dashboard loads. When something breaks at 2 AM, the on-call engineer ends up tracing a bug through a YAML config, a Jinja template, a SQL CTE and a Python script nobody has opened in six months. I think there’s a simpler way, and it has been running in production since late 2025.

One Python function fetches from any source, transforms at Rust speed and writes to any destination, with Dagster orchestrating it and pytest enforcing the contract.

— working rule
01 · The fragmentation tax

SQL wasn’t built for software engineering, and at scale that really starts to hurt.

SQL has no functions you can import, no type system you can rely on and no standard way to write a test. Every warehouse has its own dialect, and a UDF written for one warehouse won’t work on another. Moving from Snowflake to BigQuery means rewriting your transformation layer, which to me is just vendor lock-in dressed up as a query language.

Capability SQL + dbt stack Polars + Dagster + pytest stack
Languages 3–4 SQL, Jinja, YAML, Python (for edges) 1 Python
Testing dbt tests run against warehouse compute, after deploy pytest runs locally in seconds, enforceable in CI before merge
Data source portability Locked to the warehouse dialect, so a migration means rewriting transforms Same API for S3, GCS, Azure, Delta, Iceberg, Parquet, databases
Modularity Jinja macros, no real functions or classes Python functions, classes, packages, importable across projects
AI code generation LLMs generate dialect-specific SQL with frequent errors LLMs generate typed Polars expressions verified against live API
Local development Requires warehouse connection or CI run Runs on a laptop, no external dependencies
Figure 1 The capability gap at a glance. The SQL stack adds tools to make up for SQL’s limits, and the Polars stack removes the need for them.
02 · One function, any source, any destination

The same API works across S3, GCS, Azure, Delta, Iceberg, and databases.

This is the advantage that’s hardest to explain to someone who has only worked inside a warehouse. SQL is locked to the engine you’re connected to, while Polars reads from any object store, any lakehouse format and any database through the same `pl.read_*()` and `pl.scan_*()` API. Switching from CSV to Parquet is one word, moving from local dev to S3 is a path prefix, and the transformation logic stays exactly the same.

The universal reader

One API, five storage backends

It’s the same syntax whether your data is in the cloud, on disk or in a lakehouse table, so the pipeline doesn’t care where the bytes live.

Cloud object stores

S3, GCS, Azure Blob

`pl.scan_parquet("s3://bucket/orders/")` or `gs://` or `az://`, the same call with a different prefix.

Lakehouse formats

Delta Lake, Apache Iceberg

`pl.scan_delta("az://container/table/")` gives you a lazy scan and a query optimizer, with no Spark cluster needed.

Local files

CSV, Parquet, JSON, Arrow, IPC

`pl.read_csv("data.csv")` becomes `pl.read_parquet("data.parquet")`, so one word changes and none of the logic does.

Databases

Postgres, MySQL, SQLite, any connector

`pl.read_database(query, connection)` pulls exactly what you need, then you transform in Rust and write anywhere.

The real win

Portability that SQL cannot match

If you migrate warehouses, you change the connection strings and leave the transformation code alone, because the pipeline logic doesn’t depend on the warehouse.

Figure 2 One way in and out for everything. Polars separates the transformation logic from the storage backend. SQL can’t do that, since every warehouse dialect is a form of lock-in.
03 · The performance boundary

Polars with streaming beats warehouse SQL on complex transforms, for a fraction of the cost.

The January 2026 benchmark by Reliable Data Engineering ran Polars 0.20.x with the streaming engine against Snowflake X-Small, on identical data, the identical cloud and identical transforms. Each test ran ten times and the median is reported. The differences are pretty clear.

Test Polars result Snowflake SQL Read it carefully
Simple aggregation (100K–100M rows) Won every size Slower 8 vCPUs at $0.34/hr vs 4 cores at $2/hr.
String manipulation (50M rows) 7.6 seconds 23.1 seconds Vectorized Rust regex vs SQL string functions, a 3× gap.
Window functions (up to 10M rows) Won small–medium Caught up at 10M+ Both have mature window implementations, so the performance converges.
Complex joins (4 tables, filtered) Competitive More mature join algorithms Polars' predicate pushdown through joins is a structural advantage.
Customer LTV (RFM + decay) Cleaner code, 40% fewer lines Nested CTEs, UDFs Complex business logic is where Polars earns its place.
Figure 3 Polars vs Snowflake SQL, January 2026. The same data, cloud setup and methodology across five test categories. Polars wins on compute-heavy transforms, and SQL catches up at scale on simple relational operations.
04 · Data contracts, enforced

pytest gives you data contracts, while dbt tests mostly give you warehouse bills.

I think the testing gap is the biggest structural advantage of the Polars stack. dbt tests run against the warehouse, which is slow and expensive, and only after the code has already been deployed. pytest runs locally in seconds for free, and you can enforce data contracts in CI before a single row reaches production.

The testing gap

Two testing models, one structural difference

dbt tests verify data at rest, while pytest verifies the logic before it runs. So the gap has less to do with how good the tools are and more with where the verification happens.

dbt tests

After the fact

These run against warehouse compute, with `not_null`, `unique` and `accepted_values`. They’re slow, expensive and run after deploy, which makes them good for monitoring data quality but bad at preventing bugs.

pytest

Before the merge

Synthetic input goes through the Polars transform and you assert on the output. It takes 3 seconds locally, runs in CI and catches logic errors before they touch any data, without a warehouse.

What pytest can enforce that dbt cannot

Business rules as code

`assert result["segment"].is_in(["VIP","Standard"]).all()` checks that every row gets a valid segment. You write the test once and it keeps enforcing the rule.

The structural truth

Verification moves left

dbt tests catch problems in production and pytest catches them in development. A bug caught in CI costs close to nothing, while one caught in the warehouse has a real cost.

Figure 4 Where the testing happens. pytest lets you verify the transformation logic before it sees real data, and dbt tests verify data quality after the fact. Both are useful, but only pytest stops the bug before it ships.
05 · The AI wrinkle

AI coding agents write Polars more reliably than SQL.

When an LLM writes SQL, it has to guess which dialect you’re using, whether your warehouse supports `QUALIFY`, whether `STRING_AGG` or `LISTAGG` is the right function and where the optimizer will place the join. When it writes a Polars expression, the API is typed and composable, and it can be checked against a live MCP server that looks up the installed version at runtime. This already exists today.

The code-generation surface

Why agents produce better Polars than SQL

The difference mostly comes down to the size of the API. Polars has a smaller, typed and versioned API, while SQL has decades of dialect drift.

SQL generation

Dialect guessing game

The LLM has to infer Snowflake vs BigQuery vs Postgres vs Redshift, and a wrong guess means a runtime error. There’s no API to validate against while it generates.

Polars generation

Live API verification

The Polars MCP server lets agents query method signatures at runtime. `pl.col("x").sum().over("y")` either type-checks or fails with a clear error.

Production evidence

200 turns, $10–15, zero manual code

One practitioner ported an entire BigQuery + dbt layer to Polars using Claude Code, and the row-level verification passed on the first try. I’d be surprised to see that with a SQL-to-SQL migration.

The 2027 bet

Agents default to Polars

Polars has an official agent skill and MCP server, and dbt doesn’t. I think the agents your team will rely on next year are already better at generating Polars than SQL.

Figure 5 Where AI writes the code. Generating SQL is a dialect-guessing game, while generated Polars can be checked against the API. Both are getting better, but only one of them has a verification layer.
06 · The honest limits

Where SQL still wins, and why you should keep it there.

No technology replaces everything, and that goes for the Polars stack too. It works best for the parts of the pipeline where SQL is the wrong tool and only gets used because there was no alternative. Below is where SQL still wins, and why I think the best teams use both.

Scenario Default to Reason
Data already in Snowflake, BigQuery or Redshift, and the transform is SELECT, GROUP BY and basic joins SQL (dbt) The optimizer sits next to the data. Extracting to Polars means paying for egress and losing that advantage.
Team is entirely analysts, no Python experience, governance and lineage are hard requirements today SQL (dbt) The best tool is the one your team can maintain, and dbt’s docs and lineage are genuinely useful.
Multi-petabyte, governed, SLAs with auditors SQL (warehouse) Auto-scaling, query history and role-based access. Governance tooling for Polars is maturing, but it isn’t turnkey yet.
Complex business logic, string processing, ML feature engineering, multi-source ETL Polars + Dagster Method chains beat CTEs, pytest beats dbt tests and Python modules beat Jinja macros.
Local dev, CI, agentic AI pipelines Polars + Dagster No warehouse needed and instant feedback, and AI agents generate better Polars than SQL.
Figure 6 The routing table. SQL wins on simple set-based work that already lives in the warehouse and fits the team. Polars wins on complex logic, multi-source I/O, testability and AI-assisted development.
Closing observation

I think the teams that switch first will build faster, spend less and hire better.

I think the SQL-versus-Polars argument will settle the way every language-versus-language argument does. Nobody reaches a consensus, builders just vote with their time. The builders who have tried both are already writing Polars for the hard parts and keeping SQL for the simple parts. What I don’t know is how many years of unnecessary fragmentation the rest of the field will pay for before it catches up.

To me the real decision is between “one unified platform” and “five tools that barely hold together,” more than between Polars and SQL. A Polars, Dagster and pytest stack gives you one language that fetches from any source, transforms at Rust speed, writes to any destination and enforces data contracts in CI before a single row reaches production. The SQL stack gives you three languages, warehouse lock-in, tests that run against production compute, and an architecture where your most complex logic lives in stored procedures nobody can debug.

I think the cost of staying with SQL is rising faster than the cost of switching. There’s still time to build an advantage on pipeline architecture, but that window won’t stay open forever. Every month a team spends debugging Jinja templates and warehouse UDFs is a month a competitor spends writing pytest data contracts and shipping with confidence.

— end —

Replace Your SQL Pipelines with Polars · A Field Note Set in Shadows Into Light Two & Cause

← Back to all posts