Deploy a dbt project to Remote Execution Agents

Learn how to make dbt project files available to Remote Execution Agents so that Cosmos can orchestrate dbt models as Airflow tasks.

This is feature is only available if you are on the

Enterprise

tier or above. See

Astro Plans and Pricing

.

Airflow 3

This feature is only available for Airflow 3.x Deployments.

Remote Execution Agents run tasks in your own Kubernetes infrastructure. Because Remote Execution mode doesn’t support astro dbt deploy, you must make your dbt project files available to agents through other methods.

This document covers two approaches for deploying dbt code to Remote Execution Agents, ordered by Astronomer’s recommendation.

Prerequisites

• A Remote Execution Deployment with at least one registered agent.
• A secrets backend and XCom backend configured for your agents.
• A dbt project in a Git repository.
• Cosmos installed in your agent client image for orchestrating dbt models.

Option 1: Include the dbt project as a Git submodule (recommended) [#option-1]

Use Git submodules to include an external dbt project repository inside your Dag repository. This approach keeps your dbt and Airflow code in separate repositories while combining them at deploy time.

This is the recommended approach because it:

• Preserves separation of ownership between dbt and Airflow teams.
• Pins the dbt project to a specific commit, giving you control over which version of dbt code runs.
• Works with GitDagBundle for automatic Dag versioning.

How it works

Your Dag repository includes the dbt project as a Git submodule. When agents fetch the Dag bundle from Git, the submodule contents are fetched alongside your Dags. Cosmos then reads the dbt project from the submodule folder to orchestrate dbt models as Airflow tasks.

Configure the Dag repository

1. Add the dbt project as a submodule
2. From your Dag repository root, run:
3. git submodule add <dbt-repository-url> dbt/<project-name>
   

4. For example, to add a dbt project called jaffle-shop:
5. git submodule add https://github.com/your-org/jaffle-shop dbt/jaffle-shop
   

6. This creates a .gitmodules file and clones the dbt repository into the dbt/jaffle-shop folder.
7. Verify the repository structure
8. Your Dag repository should look similar to the following:
9. dag-repo/
   ├── .gitmodules
   ├── dags/
   │   └── dbt_dag.py
   └── dbt/
       └── jaffle-shop/       # Git submodule
           ├── models/
           ├── seeds/
           └── dbt_project.yml
   

10. Create a Cosmos Dag
11. Create a Dag that uses Cosmos to orchestrate the dbt project. Set the dbt_project_path to reference the submodule location within the Dag bundle:
12. from cosmos import DbtDag, ProjectConfig, ProfileConfig, ExecutionConfig
    from pathlib import Path
    from datetime import datetime
    
    dbt_dag = DbtDag(
        project_config=ProjectConfig(
            dbt_project_path=Path("/usr/local/airflow/dags/dbt/jaffle-shop"),
        ),
        profile_config=ProfileConfig(
            profile_name="jaffle_shop",
            target_name="dev",
            profiles_yml_filepath=Path("/usr/local/airflow/dags/dbt/jaffle-shop/profiles.yml"),
        ),
        execution_config=ExecutionConfig(
            dbt_executable_path="/home/astro/dbt_venv/bin/dbt",
        ),
        schedule="@daily",
        start_date=datetime(2024, 1, 1),
        catchup=False,
        dag_id="dbt_jaffle_shop",
    )
    

13. The exact path depends on your GitDagBundle configuration. If your bundle uses a subdir parameter, adjust the path accordingly.

14. Commit and push
15. git add .gitmodules dbt/jaffle-shop dags/dbt_dag.py
    git commit -m "Add dbt project as submodule with Cosmos DAG"
    git push
    

Configure agents to fetch submodules

By default, GitDagBundle does not recursively fetch Git submodules. You must configure the git_conn_id connection to include submodule support and ensure the agent can authenticate to the submodule repository.

If the dbt submodule repository uses the same authentication as the Dag repository, configure the Dag bundle with the same git_conn_id. If the submodule is in a public repository, no additional authentication is required.

For private submodule repositories that require separate credentials, configure an additional Git connection in your secrets backend or values.yaml.

Update the dbt submodule

When the dbt team pushes new changes, update the submodule reference in your Dag repository:

git submodule update --remote dbt/jaffle-shop
git add dbt/jaffle-shop
git commit -m "Update jaffle-shop submodule to latest"
git push

Agents pick up the change on the next GitDagBundle refresh cycle.

For more information about working with Git submodules in an Astro context, see Use Git submodules with an Astro project.

Option 2: Include dbt code in the agent client image [#option-2]

Build the dbt project directly into your custom agent client image. This approach works well when dbt code changes infrequently and you want a self-contained image.

1. Create a custom Dockerfile
2. Extend the base agent image to include your dbt project and dependencies:
3. FROM images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.1-3-python-3.12-astro-agent-1.2.0
   
   # Install dbt into a virtual environment
   RUN python -m venv /home/astro/dbt_venv && \
       source /home/astro/dbt_venv/bin/activate && \
       pip install --no-cache-dir dbt-postgres && \
       deactivate
   
   # Install Cosmos
   RUN pip install astronomer-cosmos
   
   # Copy dbt project into the image
   COPY dbt/jaffle-shop /usr/local/airflow/dbt/jaffle-shop
   

4. Build and push the image
5. docker build -f Dockerfile.client -t your-registry.example.com/custom-agent:1.0.0 .
   docker push your-registry.example.com/custom-agent:1.0.0
   

6. Update the Helm values
7. Reference the custom image in your values.yaml:
8. workers:
     - name: default-worker
       image: your-registry.example.com/custom-agent:1.0.0
   
   dagProcessor:
     image: your-registry.example.com/custom-agent:1.0.0
   
   triggerer:
     image: your-registry.example.com/custom-agent:1.0.0
   

9. Apply changes
10. helm upgrade astro-agent astronomer/astro-remote-execution-agent -f values.yaml
    

This approach requires rebuilding and redeploying the agent image every time dbt code changes. For teams that update dbt models frequently, consider Option 1 instead.

Compare approaches

Criteria	Git submodule	Agent image
dbt and Airflow code in separate repositories	Yes	Yes
Independent dbt deploy cycle	Partial	No
Works with GitDagBundle versioning	Yes	No
Requires image rebuild for dbt changes	No	Yes
Additional infrastructure	None	None
Best for	Most teams	Infrequent dbt changes

Related documentation

• Remote Execution overview
• Configure Dag sources
• Deploy Remote Execution project
• Deploy dbt projects to Astro
• Use Git submodules with an Astro project
• Orchestrate dbt Core jobs with Airflow and Cosmos