Dbt Architecture: Core, Cloud, and the Future With Fusion

Goal: Understand the architectural differences between dbt Core and dbt Cloud, and look ahead at the new dbt Fusion engine.

1. The Two Flavors of dbt

Historically, dbt has offered two distinct ways to run your transformations.

dbt Core (The Engine)

• What: Open-source, command-line tool.
• Cost: Free.
• Hosting: Self-hosted. You run it on your laptop, or orchestrate it via Airflow/Prefect on Kubernetes.
• Pros: Complete control, no vendor lock-in, free.
• Cons: You manage the infrastructure. No built-in scheduler or IDE.

dbt Cloud (The Platform)

• What: SaaS platform built on top of dbt Core.
• Cost: Paid (with a free developer tier).
• Hosting: Managed by dbt Labs.
• Pros:
  • Zero Setup: Web-based IDE ready to go.
  • Integrated Job Scheduler: Easy to set up production runs.
  • Hosted Docs: Share documentation with a link.
• Cons: Cost, less flexibility in underlying infrastructure.

2. The Future: dbt Fusion (2025 and beyond)

In May 2025, dbt Labs announced dbt Fusion, a complete rewrite of the underlying execution engine.

Key Improvements

1. Speed: Up to 30x faster compilation. For large projects with thousands of models, this is a game-changer.
2. Developer Experience: Better error catching before runtime.
3. Unified Vision: Fusion powers both the CLI and the Cloud IDE, bridging the gap between “local dev” and “cloud prod.”

Current Limitations (as of early 2026)

• Adapter Support: Fusion supports major warehouses (Snowflake, BigQuery, Redshift, Databricks, Postgres).
• DuckDB: Notably, DuckDB is not yet supported by Fusion. This is important for our course, as we use DuckDB for local development.

3. Our Course Approach: Back to Basics

For this Data Engineering Zoomcamp, we will use dbt Core with DuckDB.

Why?

1. Fundametals: dbt Cloud abstracts away many complexities (profiles.yml, connection management). Using Core forces you to understand exactly how dbt connects to your warehouse.
2. Local Dev: It mimics the software engineering workflow (VS Code + Terminal + Git) that Analytics Engineers use daily.
3. Cost: It’s free and runs entirely on your machine.

Note: If you prefer dbt Cloud + BigQuery, that is perfectly fine! The SQL code and modeling concepts are identical. The only difference is where you type the code (Web IDE vs VS Code) and how you run it (Button click vs Command line).

In the next post, we will set up our dbt_project.yml and learn how to configure sources.