dbt — SQL Transformation
dbt (data build tool) is the transformation layer in the Calabi pipeline. It turns raw data in your data warehouse into clean, tested, documented models using version-controlled SQL. Every dbt model is tracked in your team's Git repository, reviewed through pull requests, and executed automatically by Calabi Pipelines on schedule.
How dbt Fits in the Pipeline
Organising Your dbt Projects
Each business domain or data area is typically a separate dbt project in its own Git repository. Calabi Pipelines can run multiple projects in sequence — for example, staging models first, then business-layer marts.
A common project structure looks like this:
| Project | Target Schema | Purpose |
|---|---|---|
staging | staging | Raw-to-clean models — type casting, deduplication, renaming |
core | core | Intermediate entities — customers, orders, sessions |
reporting | reporting | Final fact and dimension tables for BI consumption |
Your projects, your naming. The names above are illustrative. Your team decides how many dbt projects to create, what to call them, and which schemas they write to. Calabi Pipelines simply runs whatever
dbt runanddbt testcommands you configure per project.
Example: Multi-Project Git Layout
git/
├── dbt-staging/ ← git repo #1 — raw to clean
│ ├── dbt_project.yml
│ ├── models/
│ │ └── staging/
│ └── tests/
├── dbt-core/ ← git repo #2 — business entities
│ ├── dbt_project.yml
│ └── models/
│ ├── intermediate/
│ └── marts/
└── dbt-reporting/ ← git repo #3 — BI-ready aggregates
├── dbt_project.yml
└── models/
└── reporting/
Calabi clones each repo at pipeline runtime, injects warehouse credentials from secrets, and runs dbt run + dbt test in the configured order.
Running dbt via Calabi Pipelines
dbt runs are triggered by Calabi Pipelines DAG tasks — no manual execution needed in production. A typical DAG task looks like:
# Run models
dbt run --project-dir /dbt/your-project --profiles-dir /dbt --target prod
# Test models (gates downstream tasks on pass)
dbt test --project-dir /dbt/your-project --profiles-dir /dbt --target prod
If tests fail, Calabi Pipelines halts the DAG and alerts your team — downstream dashboards and AI queries are never served stale or broken data.
AI Agent: dbt Tools
The Calabi AI Agent can scaffold and commit dbt changes on your behalf:
"Create a staging model for the raw orders table"
→ Agent generates SQL, creates a branch, opens a merge request
"Add a uniqueness test on order_id in the core orders model"
→ Agent writes the YAML test definition, pushes to your repo, opens MR