Replace Your SQL Pipelines with Polars
Your transformation stack is probably four tools, three languages and a warehouse you can’t really leave. A Polars, Dagster and pytest stack gives you one language, any data source in and out, and data contracts you can enforce in CI before a single row reaches production.
Every data team I know runs a transformation stack that nobody really designed on purpose. There’s Fivetran for ingestion, dbt for SQL models inside the warehouse, Python scripts for the transforms SQL can’t express, and pytest duct-taped to dbt for testing. So the pipeline touches five tools before a dashboard loads. When something breaks at 2 AM, the on-call engineer ends up tracing a bug through a YAML config, a Jinja template, a SQL CTE and a Python script nobody has opened in six months. I think there’s a simpler way, and it has been running in production since late 2025.
One Python function fetches from any source, transforms at Rust speed and writes to any destination, with Dagster orchestrating it and pytest enforcing the contract.
— working ruleSQL wasn’t built for software engineering, and at scale that really starts to hurt.
SQL has no functions you can import, no type system you can rely on and no standard way to write a test. Every warehouse has its own dialect, and a UDF written for one warehouse won’t work on another. Moving from Snowflake to BigQuery means rewriting your transformation layer, which to me is just vendor lock-in dressed up as a query language.
| Capability | SQL + dbt stack | Polars + Dagster + pytest stack |
|---|---|---|
| Languages | 3–4 SQL, Jinja, YAML, Python (for edges) | 1 Python |
| Testing | dbt tests run against warehouse compute, after deploy | pytest runs locally in seconds, enforceable in CI before merge |
| Data source portability | Locked to the warehouse dialect, so a migration means rewriting transforms | Same API for S3, GCS, Azure, Delta, Iceberg, Parquet, databases |
| Modularity | Jinja macros, no real functions or classes | Python functions, classes, packages, importable across projects |
| AI code generation | LLMs generate dialect-specific SQL with frequent errors | LLMs generate typed Polars expressions verified against live API |
| Local development | Requires warehouse connection or CI run | Runs on a laptop, no external dependencies |
The same API works across S3, GCS, Azure, Delta, Iceberg, and databases.
This is the advantage that’s hardest to explain to someone who has only worked inside a warehouse. SQL is locked to the engine you’re connected to, while Polars reads from any object store, any lakehouse format and any database through the same `pl.read_*()` and `pl.scan_*()` API. Switching from CSV to Parquet is one word, moving from local dev to S3 is a path prefix, and the transformation logic stays exactly the same.
One API, five storage backends
It’s the same syntax whether your data is in the cloud, on disk or in a lakehouse table, so the pipeline doesn’t care where the bytes live.
Cloud object stores
S3, GCS, Azure Blob
`pl.scan_parquet("s3://bucket/orders/")` or `gs://` or `az://`, the same call with a different prefix.
Lakehouse formats
Delta Lake, Apache Iceberg
`pl.scan_delta("az://container/table/")` gives you a lazy scan and a query optimizer, with no Spark cluster needed.
Local files
CSV, Parquet, JSON, Arrow, IPC
`pl.read_csv("data.csv")` becomes `pl.read_parquet("data.parquet")`, so one word changes and none of the logic does.
Databases
Postgres, MySQL, SQLite, any connector
`pl.read_database(query, connection)` pulls exactly what you need, then you transform in Rust and write anywhere.
The real win
Portability that SQL cannot match
If you migrate warehouses, you change the connection strings and leave the transformation code alone, because the pipeline logic doesn’t depend on the warehouse.
Polars with streaming beats warehouse SQL on complex transforms, for a fraction of the cost.
The January 2026 benchmark by Reliable Data Engineering ran Polars 0.20.x with the streaming engine against Snowflake X-Small, on identical data, the identical cloud and identical transforms. Each test ran ten times and the median is reported. The differences are pretty clear.
| Test | Polars result | Snowflake SQL | Read it carefully |
|---|---|---|---|
| Simple aggregation (100K–100M rows) | Won every size | Slower | 8 vCPUs at $0.34/hr vs 4 cores at $2/hr. |
| String manipulation (50M rows) | 7.6 seconds | 23.1 seconds | Vectorized Rust regex vs SQL string functions, a 3× gap. |
| Window functions (up to 10M rows) | Won small–medium | Caught up at 10M+ | Both have mature window implementations, so the performance converges. |
| Complex joins (4 tables, filtered) | Competitive | More mature join algorithms | Polars' predicate pushdown through joins is a structural advantage. |
| Customer LTV (RFM + decay) | Cleaner code, 40% fewer lines | Nested CTEs, UDFs | Complex business logic is where Polars earns its place. |
pytest gives you data contracts, while dbt tests mostly give you warehouse bills.
I think the testing gap is the biggest structural advantage of the Polars stack. dbt tests run against the warehouse, which is slow and expensive, and only after the code has already been deployed. pytest runs locally in seconds for free, and you can enforce data contracts in CI before a single row reaches production.
Two testing models, one structural difference
dbt tests verify data at rest, while pytest verifies the logic before it runs. So the gap has less to do with how good the tools are and more with where the verification happens.
dbt tests
After the fact
These run against warehouse compute, with `not_null`, `unique` and `accepted_values`. They’re slow, expensive and run after deploy, which makes them good for monitoring data quality but bad at preventing bugs.
pytest
Before the merge
Synthetic input goes through the Polars transform and you assert on the output. It takes 3 seconds locally, runs in CI and catches logic errors before they touch any data, without a warehouse.
What pytest can enforce that dbt cannot
Business rules as code
`assert result["segment"].is_in(["VIP","Standard"]).all()` checks that every row gets a valid segment. You write the test once and it keeps enforcing the rule.
The structural truth
Verification moves left
dbt tests catch problems in production and pytest catches them in development. A bug caught in CI costs close to nothing, while one caught in the warehouse has a real cost.
AI coding agents write Polars more reliably than SQL.
When an LLM writes SQL, it has to guess which dialect you’re using, whether your warehouse supports `QUALIFY`, whether `STRING_AGG` or `LISTAGG` is the right function and where the optimizer will place the join. When it writes a Polars expression, the API is typed and composable, and it can be checked against a live MCP server that looks up the installed version at runtime. This already exists today.
Why agents produce better Polars than SQL
The difference mostly comes down to the size of the API. Polars has a smaller, typed and versioned API, while SQL has decades of dialect drift.
SQL generation
Dialect guessing game
The LLM has to infer Snowflake vs BigQuery vs Postgres vs Redshift, and a wrong guess means a runtime error. There’s no API to validate against while it generates.
Polars generation
Live API verification
The Polars MCP server lets agents query method signatures at runtime. `pl.col("x").sum().over("y")` either type-checks or fails with a clear error.
Production evidence
200 turns, $10–15, zero manual code
One practitioner ported an entire BigQuery + dbt layer to Polars using Claude Code, and the row-level verification passed on the first try. I’d be surprised to see that with a SQL-to-SQL migration.
The 2027 bet
Agents default to Polars
Polars has an official agent skill and MCP server, and dbt doesn’t. I think the agents your team will rely on next year are already better at generating Polars than SQL.
Where SQL still wins, and why you should keep it there.
No technology replaces everything, and that goes for the Polars stack too. It works best for the parts of the pipeline where SQL is the wrong tool and only gets used because there was no alternative. Below is where SQL still wins, and why I think the best teams use both.
| Scenario | Default to | Reason |
|---|---|---|
| Data already in Snowflake, BigQuery or Redshift, and the transform is SELECT, GROUP BY and basic joins | SQL (dbt) | The optimizer sits next to the data. Extracting to Polars means paying for egress and losing that advantage. |
| Team is entirely analysts, no Python experience, governance and lineage are hard requirements today | SQL (dbt) | The best tool is the one your team can maintain, and dbt’s docs and lineage are genuinely useful. |
| Multi-petabyte, governed, SLAs with auditors | SQL (warehouse) | Auto-scaling, query history and role-based access. Governance tooling for Polars is maturing, but it isn’t turnkey yet. |
| Complex business logic, string processing, ML feature engineering, multi-source ETL | Polars + Dagster | Method chains beat CTEs, pytest beats dbt tests and Python modules beat Jinja macros. |
| Local dev, CI, agentic AI pipelines | Polars + Dagster | No warehouse needed and instant feedback, and AI agents generate better Polars than SQL. |
I think the teams that switch first will build faster, spend less and hire better.
I think the SQL-versus-Polars argument will settle the way every language-versus-language argument does. Nobody reaches a consensus, builders just vote with their time. The builders who have tried both are already writing Polars for the hard parts and keeping SQL for the simple parts. What I don’t know is how many years of unnecessary fragmentation the rest of the field will pay for before it catches up.
To me the real decision is between “one unified platform” and “five tools that barely hold together,” more than between Polars and SQL. A Polars, Dagster and pytest stack gives you one language that fetches from any source, transforms at Rust speed, writes to any destination and enforces data contracts in CI before a single row reaches production. The SQL stack gives you three languages, warehouse lock-in, tests that run against production compute, and an architecture where your most complex logic lives in stored procedures nobody can debug.
I think the cost of staying with SQL is rising faster than the cost of switching. There’s still time to build an advantage on pipeline architecture, but that window won’t stay open forever. Every month a team spends debugging Jinja templates and warehouse UDFs is a month a competitor spends writing pytest data contracts and shipping with confidence.
— end —