dbt

dbt (data build tool) is the transformation layer of the ELT pattern: it takes SQL written against raw warehouse tables, compiles it into a directed graph of models, and runs it inside the warehouse itself. dbt does not move data and it does not store data - it is a compiler and a runner for SQL that knows about dependencies, tests, and documentation.

Models and ref()

A model is a .sql file that selects data. The {{ ref('other_model') }} macro is the whole point: it names a dependency, so dbt builds the graph, runs models in dependency order, and generates the lineage documentation from the same structure that executes. Hard-coded table names instead of ref() are how the graph silently breaks - the model still compiles, but dbt can no longer order or document it correctly.

Materializations

How a model persists decides its cost profile:

• view - default; no storage cost, always current, but the query re-runs on every read. Cheap to define, expensive to query at scale.
• table - rebuilt fully on every run; simple and predictable, but wasteful for large append-only data.
• incremental - inserts or merges only new and changed rows on each run; the cost profile of a table with the freshness of a view, at the price of correctly written incremental logic (is_incremental(), unique_key, and the failure mode of a filter that misses late-arriving rows).
• ephemeral - compiled into its dependents as a CTE; no warehouse object at all.

The wrong materialization is the most common dbt performance problem - a view on a heavy model queried by every dashboard, or a full-table rebuild of a billion-row fact every hour when incremental would do.

Tests, sources, and snapshots

• Tests run as part of the build: unique, not_null, accepted_values, relationships are the built-ins; custom tests and dbt_utils extend them. dbt build runs tests in the graph order and stops downstream models on failure - the test is a gate, not a report.
• Sources declare the raw inputs with their freshness expectations - the contract between what the pipeline loaded and what dbt expects it to look like.
• Snapshots implement slowly-changing-dimension type 2 tracking: a snapshot records a row’s history as dbt_valid_from/dbt_valid_to windows whenever a tracked column changes, so “what did this customer look like in March” is answerable without a separate audit table.

dbt v1 versus v2 - and the platform

dbt v2 is the current generation: the Python runtime of the 1.x series was replaced by a Rust-based engine built on the same foundation as dbt’s Fusion engine. Installing dbt today gives you the Fusion distribution by default; dbt-core 2.0 remains the Apache-2.0 open-source runtime on the same codebase. The authoring model is unchanged - models, ref(), tests, and snapshots work the same way - but the internals (parser performance, language validation, artifacts) are much faster.

For v1 projects, dbt Core v1.12 is the migration bridge: it is the last actively supported v1 line (support through July 2027) and ships an opt-in --use-v2-parser flag that delegates parsing to the v2 Rust parser, so you can catch compatibility issues before switching engines. v1.11 is critical-support only; v1.10 and below are deprecated.

The dbt platform (formerly dbt Cloud) adds the scheduler, IDE, and managed infrastructure around the engine. The model language is the same; the difference is who runs the runner - which is a procurement and lock-in question more than a technical one.

Failure modes and diagnostics

• Incremental models that go stale. An is_incremental() filter that misses late or restated rows silently drifts - the fix is a periodic full-refresh or a model materialized to handle restatement.
• Broken ref() chains. Hard-coded names or a dropped model leave downstream models reading stale data without erroring - the lineage graph in dbt docs shows the gap.
• Test failures that do not fail the run. dbt run skips tests; only dbt build gates on them - a pipeline that runs run alone is testing nothing.
• Compilation cost at scale. Thousands of models make parse time and dbt docs generation noticeably slow - partial parsing exists for this reason.

What this is not for

dbt is not an orchestrator - it runs its own DAG of models, but it does not schedule across pipelines, manage external dependencies, or handle non-SQL work; that is Airflow’s job, and dbt run is typically a task inside one. It is not an ingestion tool - it does not pull data out of sources. And it is not a semantic layer: it produces tables, not a governed metrics API - that is what the semantic-layer products on top of it are for.