Deploy an Astro project as an image

Deploy a complete Astro project to a Deployment as a Docker image.

In a full deploy, the Astro CLI takes every file in your Astro project to builds them into a Docker image. This includes your Dockerfile, dags, plugins, and all Python and OS-level packages. The CLI then deploys the image to all Airflow components in a Deployment.

Use this document to learn how full deploys work and how to manually push your Astro project to a Deployment. For production environments, Astronomer recommends automating all code deploys with CI/CD. See Choose a CI/CD strategy.

See dags-only deploys to learn more about how to deploy your dags and images separately.

Prerequisites

• The Astro CLI is installed in an empty directory. If you’re using an Apple M1 system with Astro Runtime 6.0.4 or later for local development, you must install Astro CLI 1.4.0 or later to deploy to Astro.
• An Astro Workspace with at least one Deployment.
• An Astro project.

Step 1: Authenticate to Astro

Run the following command to authenticate to Astro:

wordWrap

astro login

After running this command, you are prompted to open your web browser and log in to the Astro UI. After you complete this login, you are automatically authenticated to the CLI.

If you have API credentials set as OS-level environment variables on your local machine, you can deploy directly to Astro without needing to manually authenticate. This setup is required for automating code deploys with

CI/CD

.

Step 2: Push your Astro project to an Astro Deployment

To deploy your Astro project, run:

wordWrap

astro deploy

This command returns a list of Deployments available in your Workspace and prompts you to pick one.

After you select a Deployment, the CLI parses your dags and runs a suite of pytests to ensure that they don’t contain basic errors. This testing process is equivalent to running astro dev parse and astro dev pytest in a local Airflow environment. If any of your dags fail this testing process, the deploy to Astro also fails. To force a deploy even if your project has errors, you can run astro deploy --force. For more information about using pytests, see Troubleshoot your local Airflow environment and Testing Airflow dags.

If your code passes the testing phase, the Astro CLI deploys your project in two separate, simultaneous processes:

• The Astro CLI uploads your dags directory to Astronomer-hosted blob storage. Your Deployment downloads the dags from the blob storage and applies the code to all of its running Airflow containers.
• The Astro CLI builds all other project files into a Docker image and deploys this to an Astronomer-hosted Docker registry. The Deployment then applies the image to all of its running Airflow containers.

See What happens during a project deploy for a more detailed description of how the Astro CLI and Astro work together to deploy your code.

If you use Docker Desktop, ensure that the Use containerd for pulling and storing images setting is turned off. Otherwise, you might receive errors when you run astro deploy such as:

wordWrap

Push access denied, repository does not exist or may require authorization: server message: insufficient_scope: authorization failed
# or
Unable to find image 'barren-ionization-0185/airflow:latest' locally
Error response from daemon: pull access denied for barren-ionization-0185/airflow, repository does not exist or may require 'docker login'

Step 3: Validate your changes

When you start a code deploy to Astro, the status of the Deployment is DEPLOYING until it is determined that the underlying Airflow components are running the latest version of your code. During this time, you can hover over the status indicator to determine whether your entire Astro project or only dags were deployed .

When the deploy completes, the Docker Image field in the Astro UI are updated depending on the type of deploy you completed.

• The Docker Image field displays a unique identifier generated by a Continuous Integration (CI) tool or a timestamp generated by the Astro CLI after you complete an image deploy.

To confirm a deploy was successful, verify that the running versions of your Docker image and dag bundle have been updated.

1. In the Astro UI, select a Workspace and then select a Deployment.
2. Review the information in the Docker Image and Dag bundle version field to determine the Deployment code version.

What happens during a project deploy

Hosted execution mode

Read this section for a more detailed description of the project deploy process to a Hosted execution mode Airflow 3 Deployment.

Your Deployment uses the following components to process your code deploy:

• A proprietary operator for deploying Docker images to your Airflow containers
• An Astro bundle backend for retrieving dags and non-dag bundles from versioned storage
• A blob storage container hosted by Astronomer

When you run astro deploy, the Astro CLI deploys all non-dag files in your project as an image to an Astronomer-hosted Docker registry. The proprietary operator pulls the images from a Docker registry, then updates the running image for all Airflow containers in your Deployment. Dag changes are deployed through a separate and simultaneous process.

For Airflow 3 Deployments with dag deploys enabled, the Astro bundle backend ensures dag versioning and dag rollbacks work out of the box. If dag deploys are disabled, the dags are stored in the image and can only be updated by image deploys. The Astro bundle backend also supports bundle deploys, enabling deployment of non-dag bundles such as astro dbt deploy. The backend downloads these bundles and mounts them at the configured path in the Airflow containers.

This process is different if your Deployment has dag-only deploys disabled. See

Enable/disable dag-only deploys on a Deployment

for how the process changes when dag-only deploys are disabled.

Remote execution mode

For Remote execution mode Deployments, dag deploys are always disabled, and you must configure either the LocalDagBundle or the GitDagBundle so that Remote Execution Agents in your environment can access your dags whether they are locally stored or in a git repository.

LocalDagBundle is the default Dag bundle type for the dagBundleConfigList config option but you can alternatively configure a git connection with GitDagBundle for extended versioning capabilities. See Dag bundles for more information.

How Deployments handle code deploys

After a Deployment receives an image deploy, Astro gracefully terminates all of it’s Airflow component containers, which means the existing worker Pods are allowed to finish their running tasks but the new worker Pods run your new code. For dag-only deploys, the Airflow component containers, including any workers or Kubernetes worker Pods that are currently running tasks, are preserved.

If you deploy code to a Deployment that is running a previous version of your code, then the following happens:

• Tasks that are running continue to run on existing workers and are not interrupted unless the task does not complete within 24 hours of the code deploy.

• One or more new workers are created alongside your existing workers and immediately start executing scheduled tasks based on your latest code.

  These new workers execute downstream tasks of dag runs that are in progress. For example, if you deploy to Astronomer when Task A of your dag is running, Task A continues to run on an old Celery worker. If Task B and Task C are downstream of Task A, they are both scheduled on new Celery workers running your latest code.

  This means that dag runs could fail due to downstream tasks running code from a different source than their upstream tasks. Dag runs that fail this way need to be fully restarted from the Airflow UI so that all tasks are executed based on the same source code.

Astronomer sets a grace period of 24 hours for all workers to allow running tasks to continue executing. This grace period is not configurable. If a task does not complete within 24 hours, its worker is terminated. Airflow marks the task as a zombie and it retries according to the task’s retry policy. This is to ensure that our team can reliably upgrade and maintain Astro as a service.

If you want to force long-running tasks to terminate sooner than 24 hours, specify an execution_timeout in your dag’s task definition.

Alternatively, if you find you have many long-running tasks that exceed 24 hours, you might want to refactor your dags using deferrable operators, as Astro does not guarantee reliability for tasks with a duration that exceeds 24 hours.

Deploy a prebuilt Docker image

By default, running astro deploy with the Astro CLI builds your Astro project into a Docker image and deploys it to Astro. In some cases, you might want to skip the build step and deploy a prebuilt Docker image instead.

Deploying a prebuilt Docker image allows you to:

• Test a single Docker image across Deployments instead of rebuilding it each time.
• Reduce the time it takes to deploy. If your Astro project has a number of packages that take a long time to install, it can be more efficient to build it separately.
• Specify additional mounts and arguments in your project, which is required for setups such as installing Python packages from private sources.

To deploy your Astro project as a prebuilt Docker image:

1. Run docker build from an Astro project directory or specify the command in a CI/CD pipeline. This Docker image must be based on Astro Runtime and be available in a local Docker registry. If you run this command on an Apple M1 computer or on a computer with an ARM64 processor, you must specify --platform=linux/amd64 or else the deploy will fail. Astro Deployments require an AMD64-based image and do not support ARM64 architecture.

2. (Optional) Test your Docker image in a local Airflow environment by adding the --image-name <image-name> flag to any of the following commands:

   • astro dev start
   • astro dev restart
   • astro dev parse
   • astro dev pytest

3. Run astro deploy --image-name <image-name> or specify the command in a CI/CD pipeline.

If you have dag-only deploys enabled, you can also use the --image flag to deploy a prebuilt image without also deploying your dags folder. Use astro deploy --image-name <image-name> --image.

For more information about the --image-name option, see the CLI command reference.

If you build an AMD64-based image and run astro deploy from an Apple M1 computer, you might see a warning in your terminal. You can ignore the warning.

wordWrap

WARNING: The requested image's platform (linux/amd64) does not match the detected host platform
(linux/arm64/v8) and no specific platform was requested

Related documentation

• Choose a CI/CD Strategy for deploying code to Astro
• Develop your project
• Set environment variables