Use this file to discover all available pages before exploring further.
While you can deploy your Apache Airflow code to Astro using the Astro GitHub integration, the Astro CLI, or by configuring a CI/CD pipeline, your organization might prefer to use the Astro API to deploy code. This is because using the Astro API has very few dependencies, making it compatible with almost all CI/CD environments and security requirements.If the Astro API has access to your Astro project files, you can use the deploy endpoints in the Astro API to complete either a complete project deploy, image-only deploy or dag-only deploy. You can then implement scripts to automate deploys as an alternative to using the Astro CLI or the Astro GitHub integration.This best practice guide first walks through the steps that are necessary to deploy code to Astro using the Astro API for three different code deploy methods:
A complete project deploy
An image-only deploy
A dags-only deploy
Then, the guide shows you the recommended way to combine these three automated processes to create a script with conditional logic that can automatically deploy code to Astro, depending on which types of files change in your Astro project. These examples are bash scripts that use Docker to build the image. You can also use a different scripting language, like Python, instead of bash.
If you have dag-only deploys enabled, you can create scripts for complete project deploys, dag-only deploys, or image-only deploys. The following sections include a step by step description of the workflow followed by a bash script that executes the workflow.See Deploy code to Astro for more information about the different ways to update your dags and Airflow images.
The following steps describe the different actions that the script performs to deploy a complete Astro project. Refer to What happens during a project deploy to learn the details about how a project deploy builds and deploys a new Astro image along with deploying dags.
Make a POST request to the Deploy endpoint to create a new deploy object. In your call, specify type as IMAGE_AND_DAG. Store the values for id, imageRepository, imageTag, and dagsUploadURL that are returned in the response to use in the following steps.
This action creates an object that represents the intent to deploy code to a Deployment. See the Astro API documentation for request examples.
Replace the<image-registry-hostname>value in the following steps with the hostname returned by the API forimageRepository.
Authenticate to Astronomer’s image registry using your Astro API token:
Push the image using the imageRepository and imageTag values that you retrieved in Step 1.
docker push <imageRepository>:<imageTag>
Create a tar.gz file of your Astro project dags folder:
tar -cvzf <path-to-create-tar-file>/dags.tar.gz "dags"
Make sure to clean up thedags.tar.gzfile after uploading.
Upload the tar.gz file by making a PUT call using the dagsUploadURL that you retrieved in Step 1. In this call, it is mandatory to pass the x-ms-blob-type as BlockBlob. Then, save the x-ms-version-id from the response header.
Using your deploy id, make a request to finalize the deploy. See Astro API documentation for more information about formatting the API request.
On Success, your dags have successfully uploaded and a x-ms-version-id of the dags tarball is generated in the response headers. Pass this x-ms-version-id in the requested body to finish your updates.
It might take a few minutes for the changes to update in your Deployment.
Complete project deploy script
#!/bin/bashset -ex# Prerequisites: Set variablesORGANIZATION_ID=<set organization id>DEPLOYMENT_ID=<set deployment id>ASTRO_API_TOKEN=<set api token>ASTRO_PROJECT_PATH=<set path to your Astro project># Step 1: Initialize deployecho -e "Initiating Deploy Process for deployment $DEPLOYMENT_ID\n"CREATE_DEPLOY=$(curl --location --request POST "https://api.astronomer.io/v1/organizations/$ORGANIZATION_ID/deployments/$DEPLOYMENT_ID/deploys" \--header "X-Astro-Client-Identifier: script" \--header "Content-Type: application/json" \--header "Authorization: Bearer $ASTRO_API_TOKEN" \--data '{"type": "IMAGE_AND_DAG"}' | jq '.')DEPLOY_ID=$(echo $CREATE_DEPLOY | jq -r '.id')REPOSITORY=$(echo $CREATE_DEPLOY | jq -r '.imageRepository')TAG=$(echo $CREATE_DEPLOY | jq -r '.imageTag')DAGS_UPLOAD_URL=$(echo $CREATE_DEPLOY | jq -r '.dagsUploadUrl')# Step 2: Log in to Dockerdocker login <image-registry-hostname> -u cli -p $ASTRO_API_TOKENecho -e "\nBuilding Docker image $REPOSITORY:$TAG for $DEPLOYMENT_ID from $ASTRO_PROJECT_PATH"# Step 3: Build imagedocker build -t $REPOSITORY:$TAG --platform=linux/amd64 $ASTRO_PROJECT_PATH# Step 4: Push imageecho -e "\nPushing Docker image $REPOSITORY:$TAG to $DEPLOYMENT_ID"docker push $REPOSITORY:$TAG# Step 5: Create tar fileecho -e "\nCreating a dags tar file from $ASTRO_PROJECT_PATH/dags and stored in $ASTRO_PROJECT_PATH/dags.tar.gz\n"cd $ASTRO_PROJECT_PATHtar -cvzf "$ASTRO_PROJECT_PATH/dags.tar.gz" "dags"# Step 6: Upload dags tar fileecho -e "\nUploading tar file $ASTRO_PROJECT_PATH/dags.tar.gz\n"VERSION_ID=$(curl -i --request PUT $DAGS_UPLOAD_URL \--header 'x-ms-blob-type: BlockBlob' \--header 'Content-Type: application/x-gtar' \--upload-file "$ASTRO_PROJECT_PATH/dags.tar.gz" | grep x-ms-version-id | awk -F': ' '{print $2}')VERSION_ID=$(echo $VERSION_ID | sed 's/\r//g') # Remove unexpected carriage return charactersecho -e "\nTar file uploaded with version: $VERSION_ID\n"# Step 7: Finalizing DeployFINALIZE_DEPLOY=$(curl --location --request POST "https://api.astronomer.io/v1/organizations/$ORGANIZATION_ID/deployments/$DEPLOYMENT_ID/deploys/$DEPLOY_ID/finalize" \--header "X-Astro-Client-Identifier: script" \--header "Content-Type: application/json" \--header "Authorization: Bearer $ASTRO_API_TOKEN" \--data '{"dagTarballVersion": "'$VERSION_ID'"}')ID=$(echo $FINALIZE_DEPLOY | jq -r '.id')if [[ "$ID" != null ]]; then echo -e "\nDeploy is Finalized. Image and dag changes for deployment $DEPLOYMENT_ID should be live in a few minutes" echo "Deployed Image tag: $TAG" echo "Deployed dag Tarball Version: $VERSION_ID"else MESSAGE=$(echo $FINALIZE_DEPLOY | jq -r '.message') if [[ "$MESSAGE" != null ]]; then echo $MESSAGE else echo "Something went wrong. Reach out to astronomer support for assistance" fifi# Cleanupecho -e "\nCleaning up the created tar file from $ASTRO_PROJECT_PATH/dags.tar.gz"rm -rf "$ASTRO_PROJECT_PATH/dags.tar.gz"
The following script allows you to update only your Dag files. Refer to Deploy dags to learn the details about how Astro deploys dags.
Make a POST request to the Deploy endpoint to create a new deploy object. In your request, specify type as DAG_ONLY. Store the values for id and dagsUploadURL that are returned in the response to use in the following steps.This action creates an object that represents the intent to deploy code to a Deployment. See the Astro API documentation for request usage and examples.
Create a tar.gz file of your Astro project dags folder:
tar -cvzf <path to create the tar file>/dags.tar.gz "dags"
Make sure to clean up thedags.tar.gzfile after uploading.
Upload the tar file by making a PUT request using the dagsUploadURL that you retrieved in Step 2. In this request, it is mandatory to pass the x-ms-blob-type as BlockBlob. Then, save the x-ms-version-id from the response header.
Using your deploy id, make a request to finalize the deploy. See Astro API documentation for more information about formatting the API request.
On Success, your dags have successfully uploaded and a x-ms-version-id of the dags tarball is generated. Pass this x-ms-version-id in the requested body to finish your updates.
It might take a few minutes for the changes to update in your Deployment.
dag-only deploy script
#!/bin/bashset -ex# Prerequisites: Set variablesORGANIZATION_ID=<set organization id>DEPLOYMENT_ID=<set deployment id>ASTRO_API_TOKEN=<set api token>ASTRO_PROJECT_PATH=<set path to your airflow project># Step 1: Initialize deployecho -e "Initiating Deploy Process for deployment $DEPLOYMENT_ID\n"CREATE_DEPLOY=$(curl --location --request POST "https://api.astronomer.io/v1/organizations/$ORGANIZATION_ID/deployments/$DEPLOYMENT_ID/deploys" \--header "X-Astro-Client-Identifier: script" \--header "Content-Type: application/json" \--header "Authorization: Bearer $ASTRO_API_TOKEN" \--data '{"type": "DAG_ONLY"}' | jq '.')DEPLOY_ID=$(echo $CREATE_DEPLOY | jq -r '.id')DAGS_UPLOAD_URL=$(echo $CREATE_DEPLOY | jq -r '.dagsUploadUrl')# Step 2: Create a tar file of Astro project dags folderecho -e "\nCreating a dags tar file from $ASTRO_PROJECT_PATH/dags and stored in $ASTRO_PROJECT_PATH/dags.tar.gz\n"cd $ASTRO_PROJECT_PATHtar -cvzf "$ASTRO_PROJECT_PATH/dags.tar.gz" "dags"# Step 3: Upload tar fileecho -e "\nUploading tar file $ASTRO_PROJECT_PATH/dags.tar.gz\n"VERSION_ID=$(curl -i --request PUT $DAGS_UPLOAD_URL \--header 'x-ms-blob-type: BlockBlob' \--header 'Content-Type: application/x-gtar' \--upload-file "$ASTRO_PROJECT_PATH/dags.tar.gz" | grep x-ms-version-id | awk -F': ' '{print $2}')VERSION_ID=$(echo $VERSION_ID | sed 's/\r//g') # Remove unexpected carriage return charactersecho -e "\nTar file uploaded with version: $VERSION_ID\n"# Step 4: Finalizing DeployFINALIZE_DEPLOY=$(curl --location --request POST "https://api.astronomer.io/v1/organizations/$ORGANIZATION_ID/deployments/$DEPLOYMENT_ID/deploys/$DEPLOY_ID/finalize" \--header "X-Astro-Client-Identifier: script" \--header "Content-Type: application/json" \--header "Authorization: Bearer $ASTRO_API_TOKEN" \--data '{"dagTarballVersion": "'$VERSION_ID'"}')ID=$(echo $FINALIZE_DEPLOY | jq -r '.id')if [[ "$ID" != null ]]; then echo -e "\nDeploy is Finalized. Dag changes for deployment $DEPLOYMENT_ID should be live in a few minutes" echo "Deployed dag Tarball Version: $VERSION_ID"else MESSAGE=$(echo $FINALIZE_DEPLOY | jq -r '.message') if [[ "$MESSAGE" != null ]]; then echo $MESSAGE else echo "Something went wrong. Reach out to astronomer support for assistance" fifi# Cleanupecho -e "\nCleaning up the created tar file from $ASTRO_PROJECT_PATH/dags.tar.gz"rm -rf "$ASTRO_PROJECT_PATH/dags.tar.gz"
The following script allows you to update only your Astro project by building and deploying a new Docker image. Refer to What happens during a project deploy to learn the details about how Astro deploys image updates.
Make a POST request to the Deploy endpoint to create a new deploy object. In your request, specify type as IMAGE_ONLY. Store the value for the DeployID that is returned. This action creates an object that represents the intent to deploy code to a Deployment. See the Astro API documentation for request usage and examples.
Push the image using the imageRepository and imageTag values that you retrieved earlier.
docker push imageRepository:imageTag
Using your deploy id, make a request to finalize the deploy. See Astro API documentation for more information about formatting the API request.
On Success, your dags have successfully uploaded and a x-ms-version-id of the dags tarball is generated. Pass this x-ms-version-id in the requested body to finish your updates.
It might take a few minutes for the changes to update in your Deployment.
Image-only deploy script
#!/bin/bash set -ex # Prerequisites: Set variables ORGANIZATION_ID=<set organization id> DEPLOYMENT_ID=<set deployment id> ASTRO_API_TOKEN=<set api token> ASTRO_PROJECT_PATH=<set path to your airflow project> # Step 1: Initialize Deploy echo -e "Initiating Deploy Process for deployment $DEPLOYMENT_ID\n" CREATE_DEPLOY=$(curl --location --request POST "https://api.astronomer.io/v1/organizations/$ORGANIZATION_ID/deployments/$DEPLOYMENT_ID/deploys" \ --header "X-Astro-Client-Identifier: script" \ --header "Content-Type: application/json" \ --header "Authorization: Bearer $ASTRO_API_TOKEN" \ --data '{ "type": "IMAGE_ONLY" }' | jq '.') DEPLOY_ID=$(echo $CREATE_DEPLOY | jq -r '.id') REPOSITORY=$(echo $CREATE_DEPLOY | jq -r '.imageRepository') TAG=$(echo $CREATE_DEPLOY | jq -r '.imageTag') # Step 2 Log in to Docker docker login <image-registry-hostname> -u cli -p $ASTRO_API_TOKEN # Step 3: Build Docker image echo -e "\nBuilding Docker image $REPOSITORY:$TAG for $DEPLOYMENT_ID from $ASTRO_PROJECT_PATH" docker build -t $REPOSITORY:$TAG --platform=linux/amd64 $ASTRO_PROJECT_PATH # Step 4: Push image echo -e "\nPushing Docker image $REPOSITORY:$TAG to $DEPLOYMENT_ID" docker push $REPOSITORY:$TAG # Step 5: Finalize Deploy FINALIZE_DEPLOY=$(curl --location --request POST "https://api.astronomer.io/v1/organizations/$ORGANIZATION_ID/deployments/$DEPLOYMENT_ID/deploys/$DEPLOY_ID/finalize" \ --header "X-Astro-Client-Identifier: script" \ --header "Content-Type: application/json" \ --header "Authorization: Bearer $ASTRO_API_TOKEN" \ --data '{}') ID=$(echo $FINALIZE_DEPLOY | jq -r '.id') if [[ "$ID" != null ]]; then echo -e "\nDeploy is Finalized. Image changes for deployment $DEPLOYMENT_ID should be live in a few minutes" echo "Deployed Image tag: $TAG" else MESSAGE=$(echo $FINALIZE_DEPLOY | jq -r '.message') if [[ "$MESSAGE" != null ]]; then echo $MESSAGE else echo "Something went wrong. Reach out to astronomer support for assistance" fi fi
You can only use complete project deploys if you have dag-only deploys disabled. Image-only and dag-only deploys do not work.See Deploy code to Astro for more information about the different ways to update your dags and Airflow images.
Create the Deploy API call using the IMAGE_AND_DAG type. This action creates an object that represents the intent to deploy code to a Deployment.
After you create the deploy, retrieve the id, imageRepository, and imageTag values using the GET deploy API call, which you need in the following steps.
(Recommended) Dynamic deploys based on files changed
In addition to manually triggering project, dag, and image deploys, you can create scripts that trigger specific code deploys depending on whether there have been changes to dags or files used to build the project image.In this section, you can read a description of the process that the following code example shows the recommended way to combine the previous scripts to trigger different code deploys depending on the files changed in your project. This example demonstrates how to combine all three types of deploys and the conditional logic for when to deploy each.
This recommended process requires that you enable dag-only deploys.
In the following code example, the script completes the following processes:
Determine whether Dag files have been changed. If only dags have changed, then the script initiates a dag-only deploy.
Make a POST request to the Deploy endpoint to create a new deploy object.
If only dags have changed, the script initiates a dags-only deploy. If you changed more files, the script builds and deploys the project image, and then completes a dag-only deploy.
The script cleans up any tar files created during the build process.
To run the script, set the different environment variables to the values for your environment.
Trigger deploys when files change code example
#!/bin/bashset -ex# Prerequisites: Set variablesORGANIZATION_ID=<set organization id>DEPLOYMENT_ID=<set deployment id>ASTRO_API_TOKEN=<set api token>ASTRO_PROJECT_PATH=<set path to your airflow project># Step 1: Determine if only dag files have changesfiles=$(git diff --name-only $(git rev-parse HEAD~1) -- .)dags_only=1for file in $files;doif [[ $file != "$ASTRO_PROJECT_PATH/dags"* ]];then echo "$file is not a dag, triggering a full image build" dags_only=0 breakfidoneDEPLOY_ID=$(echo $CREATE_DEPLOY | jq -r '.id')# Step 3: If only dags changed deploy only the dags in your 'dags' folder to your Deploymentif [ $dags_only == 1 ]then echo -e "Initiating Deploy Process for deployment $DEPLOYMENT_ID\n" CREATE_DEPLOY=$(curl --location --request POST "https://api.astronomer.io/v1/organizations/$ORGANIZATION_ID/deployments/$DEPLOYMENT_ID/deploys" \ --header "X-Astro-Client-Identifier: script" \ --header "Content-Type: application/json" \ --header "Authorization: Bearer $ASTRO_API_TOKEN" \ --data '{ "type": "DAG_ONLY" }' | jq '.') DEPLOY_ID=$(echo $CREATE_DEPLOY | jq -r '.id') # Upload dags tar file DAGS_UPLOAD_URL=$(echo $CREATE_DEPLOY | jq -r '.dagsUploadUrl') echo -e "\nCreating a dags tar file from $ASTRO_PROJECT_PATH/dags and stored in $ASTRO_PROJECT_PATH/dags.tar.gz\n" cd $ASTRO_PROJECT_PATH tar -cvxf "$ASTRO_PROJECT_PATH/dags.tar.gz" "dags" echo -e "\nUploading tar file $ASTRO_PROJECT_PATH/dags.tar.gz\n" VERSION_ID=$(curl -i --request PUT $DAGS_UPLOAD_URL \ --header 'x-ms-blob-type: BlockBlob' \ --header 'Content-Type: application/x-gtar' \ --upload-file "$ASTRO_PROJECT_PATH/dags.tar.gz" | grep x-ms-version-id | awk -F': ' '{print $2}') VERSION_ID=$(echo $VERSION_ID | sed 's/\r//g') # Remove unexpected carriage return characters echo -e "\nTar file uploaded with version: $VERSION_ID\n" # Finalizing Deploy FINALIZE_DEPLOY=$(curl --location --request POST "https://api.astronomer.io/v1/organizations/$ORGANIZATION_ID/deployments/$DEPLOYMENT_ID/deploys/$DEPLOY_ID/finalize" \ --header "X-Astro-Client-Identifier: script" \ --header "Content-Type: application/json" \ --header "Authorization: Bearer $ASTRO_API_TOKEN" \ --data '{"dagTarballVersion": "'$VERSION_ID'"}') ID=$(echo $FINALIZE_DEPLOY | jq -r '.id') if [[ "$ID" != null ]]; then echo -e "\nDeploy is Finalized. Dag changes for deployment $DEPLOYMENT_ID should be live in a few minutes" echo "Deployed dag Tarball Version: $VERSION_ID" else MESSAGE=$(echo $FINALIZE_DEPLOY | jq -r '.message') if [[ "$MESSAGE" != null ]]; then echo $MESSAGE else echo "Something went wrong. Reach out to astronomer support for assistance" fi fi # Step 4: Cleanup echo -e "\nCleaning up the created tar file from $ASTRO_PROJECT_PATH/dags.tar.gz" rm -rf "$ASTRO_PROJECT_PATH/dags.tar.gz"fi# Step 3: If any other files changed build your Astro project into a Docker image, push the image to your Deployment, and then push and dag changesif [ $dags_only == 0 ]then echo -e "Initiating Deploy Process for deployment $DEPLOYMENT_ID\n" CREATE_DEPLOY=$(curl --location --request POST "https://api.astronomer.io/v1/organizations/$ORGANIZATION_ID/deployments/$DEPLOYMENT_ID/deploys" \ --header "X-Astro-Client-Identifier: script" \ --header "Content-Type: application/json" \ --header "Authorization: Bearer $ASTRO_API_TOKEN" \ --data '{ "type": "IMAGE_AND_DAG" }' | jq '.') DEPLOY_ID=$(echo $CREATE_DEPLOY | jq -r '.id') # Build and Push Docker Image REPOSITORY=$(echo $CREATE_DEPLOY | jq -r '.imageRepository') TAG=$(echo $CREATE_DEPLOY | jq -r '.imageTag') docker login <image-registry-hostname> -u cli -p $ASTRO_API_TOKEN echo -e "\nBuilding Docker image $REPOSITORY:$TAG for $DEPLOYMENT_ID from $ASTRO_PROJECT_PATH" docker build -t $REPOSITORY:$TAG --platform=linux/amd64 $ASTRO_PROJECT_PATH echo -e "\nPushing Docker image $REPOSITORY:$TAG to $DEPLOYMENT_ID" docker push $REPOSITORY:$TAG # Upload dags tar file DAGS_UPLOAD_URL=$(echo $CREATE_DEPLOY | jq -r '.dagsUploadUrl') echo -e "\nCreating a dags tar file from $ASTRO_PROJECT_PATH/dags and stored in $ASTRO_PROJECT_PATH/dags.tar.gz\n" cd $ASTRO_PROJECT_PATH tar -cvxf "$ASTRO_PROJECT_PATH/dags.tar.gz" "dags" echo -e "\nUploading tar file $ASTRO_PROJECT_PATH/dags.tar.gz\n" VERSION_ID=$(curl -i --request PUT $DAGS_UPLOAD_URL \ --header 'x-ms-blob-type: BlockBlob' \ --header 'Content-Type: application/x-gtar' \ --upload-file "$ASTRO_PROJECT_PATH/dags.tar.gz" | grep x-ms-version-id | awk -F': ' '{print $2}') VERSION_ID=$(echo $VERSION_ID | sed 's/\r//g') # Remove unexpected carriage return characters echo -e "\nTar file uploaded with version: $VERSION_ID\n" # Finalizing Deploy FINALIZE_DEPLOY=$(curl --location --request POST "https://api.astronomer.io/v1/organizations/$ORGANIZATION_ID/deployments/$DEPLOYMENT_ID/deploys/$DEPLOY_ID/finalize" \ --header "X-Astro-Client-Identifier: script" \ --header "Content-Type: application/json" \ --header "Authorization: Bearer $ASTRO_API_TOKEN" \ --data '{"dagTarballVersion": "'$VERSION_ID'"}') ID=$(echo $FINALIZE_DEPLOY | jq -r '.id') if [[ "$ID" != null ]]; then echo -e "\nDeploy is Finalized. Image and dag changes for deployment $DEPLOYMENT_ID should be live in a few minutes" echo "Deployed Image tag: $TAG" echo "Deployed dag Tarball Version: $VERSION_ID" else MESSAGE=$(echo $FINALIZE_DEPLOY | jq -r '.message') if [[ "$MESSAGE" != null ]]; then echo $MESSAGE else echo "Something went wrong. Reach out to astronomer support for assistance" fi fi # Step 4: Cleanup echo -e "\nCleaning up the created tar file from $ASTRO_PROJECT_PATH/dags.tar.gz" rm -rf "$ASTRO_PROJECT_PATH/dags.tar.gz"fi