dbt Pipelines on GitHub Actions

A dbt project gets slow and expensive on GitHub Actions when every pull request rebuilds the entire DAG on a runner that spends most of its minutes waiting on the warehouse. The fix is a slim run: publish target/manifest.json from the main branch, select state:modified+ against that manifest with --defer, and run the job on a small warp- Linux runner with the dbt package cache and parse cache restored.

Overview

A dbt job on GitHub Actions spends its time in four places: installing the Python environment and dbt adapter, resolving packages with dbt deps, parsing and compiling the project, and executing models and tests against the warehouse. Only the middle two consume runner CPU in any real quantity. Execution is a set of queries submitted over the network, so the runner sits at low CPU while the warehouse does the work, and buying a larger machine changes nothing about how long the run takes.

That makes dbt the opposite of a compile-bound workload, and the two levers that matter are how many nodes you build and how much of the setup you cache.

WarpBuild runners register against your organization under warp- labels, so switching a dbt workflow is a change to runs-on with the rest of the steps untouched. Runners are ephemeral virtual machines, allocated per job and destroyed when the job ends, which is what keeps per-pull-request warehouse schemas from leaking state between runs. The full label list and sizes are in the cloud runners documentation.

Configuration

Two workflows carry the pattern. The main-branch workflow builds the whole project and publishes the manifest that becomes the comparison state. The pull request workflow downloads that manifest and builds only what changed.

Start with the producer:

name: dbt-main

on:
  push:
    branches: [main]
  schedule:
    - cron: "0 6 * * *"

jobs:
  build:
    runs-on: warp-ubuntu-latest-x64-2x
    env:
      DBT_PROFILES_DIR: ./ci
      DBT_ACCOUNT: ${{ vars.DBT_ACCOUNT }}
      DBT_USER: ${{ vars.DBT_PROD_USER }}
      DBT_ENV_SECRET_PASSWORD: ${{ secrets.DBT_PROD_PASSWORD }}
      DBT_TARGET_SCHEMA: analytics
    steps:
      - uses: actions/checkout@v4

      - uses: astral-sh/setup-uv@v5
        with:
          python-version: "3.12"

      - name: Restore dbt packages and parse cache
        uses: WarpBuilds/cache@v1
        with:
          path: |
            ~/.cache/uv
            dbt_packages
            target/partial_parse.msgpack
          key: ${{ runner.os }}-dbt-${{ hashFiles('uv.lock', 'package-lock.yml') }}
          restore-keys: |
            ${{ runner.os }}-dbt-

      - run: uv sync --frozen
      - run: uv run dbt deps

      - name: Build the full project
        run: uv run dbt build --target prod

      - name: Publish manifest for state comparison
        uses: actions/upload-artifact@v4
        with:
          name: dbt-manifest
          path: target/manifest.json
          retention-days: 90

Now the consumer. It fetches the manifest from the most recent successful main-branch run, selects modified nodes plus their children, and defers everything else to production:

name: dbt-pr

on:
  pull_request:

permissions:
  contents: read
  actions: read

jobs:
  slim-build:
    runs-on: warp-ubuntu-latest-x64-2x
    env:
      DBT_PROFILES_DIR: ./ci
      DBT_ACCOUNT: ${{ vars.DBT_ACCOUNT }}
      DBT_USER: ${{ vars.DBT_CI_USER }}
      DBT_ENV_SECRET_PASSWORD: ${{ secrets.DBT_CI_PASSWORD }}
      DBT_TARGET_SCHEMA: ci_pr_${{ github.event.pull_request.number }}
    steps:
      - uses: actions/checkout@v4

      - uses: astral-sh/setup-uv@v5
        with:
          python-version: "3.12"

      - name: Restore dbt packages and parse cache
        uses: WarpBuilds/cache@v1
        with:
          path: |
            ~/.cache/uv
            dbt_packages
            target/partial_parse.msgpack
          key: ${{ runner.os }}-dbt-${{ hashFiles('uv.lock', 'package-lock.yml') }}
          restore-keys: |
            ${{ runner.os }}-dbt-

      - run: uv sync --frozen
      - run: uv run dbt deps

      - name: Fetch production manifest
        continue-on-error: true
        env:
          GH_TOKEN: ${{ secrets.GITHUB_TOKEN }}
        run: |
          run_id=$(gh run list --workflow dbt-main.yml --branch main \
            --status success --limit 1 --json databaseId --jq '.[0].databaseId')
          gh run download "$run_id" --name dbt-manifest --dir prod-state

      - name: Build and test modified models
        run: |
          if [ -f prod-state/manifest.json ]; then
            uv run dbt build --select 'state:modified+' \
              --defer --favor-state --state prod-state
          else
            uv run dbt build
          fi

      - name: Publish run results
        if: always()
        uses: actions/upload-artifact@v4
        with:
          name: dbt-run-results-${{ github.run_id }}
          path: target/run_results.json

Three details in that file do the work.

dbt build runs each model's tests as soon as that model finishes, in DAG order, so a failing test stops its children instead of letting the run finish and report at the end. Selecting with state:modified+ carries the tests along, because a test attached to a selected model is selected with it.

--defer --favor-state tells dbt to resolve ref() calls for unselected nodes against the production locations recorded in the state manifest. The per-pull-request schema then holds only the models you actually changed, which keeps warehouse storage and schema cleanup small.

Credentials come in as environment variables read by profiles.yml through env_var(). dbt scrubs values of variables whose names start with DBT_ENV_SECRET_ from its logs and artifacts, per the dbt environment variable documentation, so the password never lands in run_results.json or a debug log. Warehouses that support workload identity federation let you drop the password entirely and add id-token: write to permissions instead.

Sizing

Runner size sets how fast the project parses and compiles. It does not set how fast the warehouse executes, so start small and move up only on measured evidence. Rates are from the pricing page.

2x. The default for dbt. Projects up to a few hundred models parse in seconds with the partial parse cache restored, and the run is dominated by warehouse wait. The threads setting in profiles.yml controls how many queries dbt keeps in flight, and threads are cheap to raise on 2 vCPUs because each one mostly blocks on the network.

4x. Projects in the low thousands of models, or workflows that also run dbt docs generate, which loads the full catalog into memory before writing it. Also the right size when the job runs seeds that read large CSV files off disk.

8x. The ceiling worth paying for on a dbt job, and only when a timed parse or compile phase justifies it. At that size the arithmetic is on your side: warp-ubuntu-latest-x64-8x (8 vCPU, 32 GB) costs $0.016 per minute against $0.022 per minute for the 8-core Linux larger runner (8 vCPU, 32 GB): 27 percent lower list price. GitHub list price checked on 2026-08-13, from the GitHub Actions billing reference.

The ARM64 rows carry lower rates at the same shape. Check that your adapter publishes ARM64 Linux wheels before switching, since a missing wheel makes pip compile the connector from source inside the job.

Cost model: full builds against state-selected runs

Take a project with 900 models, 300 pull request runs a month, and a nightly full build on main. Assume the full dbt build takes 34 minutes of runner wall clock and the average pull request touches 12 models, which state:modified+ expands to about 60 nodes and 5 minutes. Substitute your own numbers; the shape holds regardless.

The slim pattern plus the nightly full build comes to $10.08 a month of runner time against $40.80 for rebuilding everything on each pull request. Warehouse compute moves in the same direction and is billed separately by your warehouse provider, which is normally the larger of the two lines.

Bottlenecks

Full builds on every pull request. The default dbt build has no idea what changed, so it materializes the whole project to validate a one-model edit. State comparison is the fix, and the cost is one artifact upload per main-branch run.

A missing or stale state manifest. --state pointing at an empty directory makes state:modified+ fail, and a manifest from a run that predates a schema change selects the wrong node set. The workflow above handles the first case with continue-on-error plus a full-build fallback, so an absent artifact produces a complete run instead of an empty selection. Handle the second by publishing the manifest from every successful main-branch build, including the scheduled one, so the newest artifact is never more than a day old.

Cold dbt deps and interpreter setup. Without a cache, each job re-resolves the Python environment and re-clones every dbt package. WarpBuilds/cache is a drop-in replacement for actions/cache@v4, keyed the same way, and the cache is enabled by default on Linux runners with entries expiring 7 days after last use. Keying on hashFiles('uv.lock', 'package-lock.yml') makes the hit exact, and the restore-keys prefix salvages a partial match when one package moves. Carrying cost is visible in the price list: cache storage is $0.20 per GB-month and each write or restore operation is $0.0001, so a 300MB dbt cache costs about $0.06 a month to keep warm. The caching documentation covers the split restore and save variants.

Project parse time. dbt writes target/partial_parse.msgpack and reuses it to skip re-parsing unchanged files. Caching that file across runs is worthwhile on large projects, with one caveat: dbt discards it after a version change or a change to the environment variables the project reads, so treat a sudden jump in parse time after a dbt upgrade as expected rather than as a cache bug.

Thread count against warehouse concurrency. Threads too low leaves the DAG width unused while the warehouse idles. Threads too high queues queries inside the warehouse and adds nothing. WarpBuild CI observability reports CPU and memory from the runner agent correlated with your job logs, which is the fastest way to confirm the runner idles while the warehouse works and that the answer is a threads change instead of a bigger machine. When a profile or credential failure only reproduces inside GitHub Actions, the Action Debugger pauses the workflow and opens an SSH session on the live runner so you can run dbt debug against the real environment.

Proof

Public repositories running warp- labels are checkable evidence, and the sizing argument on this page is the one those repositories already follow: match the label to the job instead of buying one large size for everything. Bitcoin Core routes its Linux jobs across sizes in bitcoin/bitcoin's ci.yml, with lint on warp-ubuntu-latest-x64-2x and the heavier sanitizer matrix on warp-ubuntu-latest-x64-8x and warp-ubuntu-latest-x64-16x (checked on 2026-08-13). The Trigger.dev end-to-end suite runs its matrix on warp-ubuntu-latest-x64-4x and warp-windows-latest-x64-8x in triggerdotdev/trigger.dev's e2e.yml, checked on 2026-08-13. The runs-on lines are open to read.

Teams that run Python transformation code alongside dbt can apply the same cache and sizing approach from the Python on GitHub Actions page, and the selection idea generalizes past dbt: the test impact analysis guide covers the same trade for application test suites.

FAQ

Do dbt runs on GitHub Actions need a large runner?

Usually no. dbt pushes the work to your warehouse and the runner mostly submits queries and waits, so warp-ubuntu-latest-x64-2x at $0.004 per minute handles most projects. Move up a size when project parse, jinja compile, or dbt docs generate takes a visible share of the run.

How does a pull request run get the production manifest?

Upload target/manifest.json as an artifact from every successful main-branch build, then download it in the pull request job with gh run download and pass the directory to --state. When the download fails, fall back to a full dbt build so a missing manifest produces a complete run instead of an empty selection.

What does a dbt pipeline cost on WarpBuild runners?

In the worked model above, state-selected pull request runs cost $0.020 each and $6.00 a month, while the nightly full builds add $4.08. That makes runner time $10.08 a month; warehouse compute is billed separately by the warehouse provider.