Configure Deployment resources

Configure your Deployment resource settings to optimize Deployment performance.

Your Deployment resources are the computational resources Astro uses to run Airflow in the cloud. Update Deployment resource settings to optimize performance and reduce the cost of running Airflow in the cloud.

Update Airflow configurations

To update a Deployment’s Airflow configurations, you set the configurations as environment variables on Astro. See Set Airflow configurations using environment variables.

Update environment objects

You can add, update, and delete Airflow connections that were added through the Astro UI from your Deployment page. To edit a Deployment’s linked connections, click the Environment tab, and then select the connection you want to Edit. See Create connections with the Astro UI for more options.

Deployment executor

Astro supports three executors: the Astro, Celery, and Kubernetes executors. Only Celery and Kubernetes executors are available in the Apache Airflow open source project.

• Celery executor
• Kubernetes executor (Requires Astro Runtime version 8.1.0 or later)
• Astro executor (Only available for Airflow 3.x Deployments)

See Choose an executor to understand the benefits and limitations of each executor. When you’ve determined the right executor type for your Deployment, complete the steps in the following topic to update your Deployment’s executor type.

Update the Deployment executor

1. In the Astro UI, select a Workspace, click Deployments, and then select a Deployment.

2. Click the Details tab and then click Edit in the Execution section.

   

3. In the Execution section, select Astro, Celery or Kubernetes in the Executor list.

   If you’re moving from the Astro or Celery to the Kubernetes executor, all existing worker queues are deleted. Running tasks stop gracefully and all new tasks start with the selected executor.

4. Click Update Deployment.

See Configure an executor for more information about each available executor type, including how to optimize executor usage.

Configure Kubernetes Pod resources

The Kubernetes executor and KubernetesPodOperator both use Kubernetes Pods to execute tasks. While you still need to configure Pods in your dag code to define individual task environments, you can set some safeguards on Astro so that tasks in your Deployment don’t request more CPU or memory than expected.

Set safeguards by configuring default Pod limits and requests from the Astro UI. If a task requests more CPU or memory than is currently allowed in your configuration, the task fails.

By default, Astro supports a maximum KubernetesExecutor Pod size of 43 vCPU and 86 GiB of memory. If your workloads require more resources, contact Astronomer support to request larger Pod sizes for your Deployment.

To customize the resources for individual task Pods beyond the defaults, see Customize a task’s Kubernetes Pod.

To manage Kubernetes resources programmatically, you can set default Pod limits and resources with the

astro deployment create

and

astro deployment update

Astro CLI commands, or by adding the configurations to a

Deployment file

.

1. In the Astro UI, select a Workspace, click Deployments, and then select a Deployment.

2. Click the Options menu and select Edit Deployment.

   

3. In the Execution section, configure the following values:

   • CPU Quota: The maximum combined CPU usage across all running Pods on your Deployment.
   • Memory Quota: The maximum combined memory usage across all running Pods on your Deployment.
   • Default Pod Size:
     • CPU: The amount of CPUs that your tasks run with if no CPU usage is specified in their Pod configuration.
     • Memory: The amount of memory that your tasks run with if no memory usage is specified in their Pod configuration.
     • Storage: Choose the amount of ephemeral storage in GiB assigned to each pod in the Astro UI. This storage volume is transient and allows for the temporary storage and processing of data. The pod is assigned the minimum 0.25 GiB by default. The maximum possible quota is 100 GiB. Only ephemeral storage requests that are greater than the default minimum of 0.25 GiB are chargeable. Note that this feature is in Public Preview.

   For a Deployment running in a Hosted dedicated or shared cluster, the maximum possible CPU quota is 6400 vCPU and maximum Memory quota is 12800 GiB.

Astro Hosted

On Astro Hosted, Astro automatically sets resource requests equal to limits for task Pods. This ensures Pods receive a Kubernetes Guaranteed Quality of Service (QoS) class, which prevents resource contention and eviction. Because Astro uses the limit values as both requests and limits, your Pods are billed based on the CPU and Memory limits you configure, even if actual usage is lower. To avoid unexpected charges, set limits close to the resources your tasks require.

Check your Billing and usage to view your resource use and associated charges.

4. Click Update Deployment.

Workers and worker queues

Workers are the Airflow components that execute your tasks. How workers are configured depends on the executor your Deployment uses:

• Astro and Celery executors: Workers run in configurable worker queues. Each worker queue defines the worker type (size), concurrency, autoscaling limits, and ephemeral storage for its workers. You can create multiple worker queues to optimize execution environments for different types of tasks. For example, you can assign resource-intensive machine learning tasks to a queue with larger workers, while running lightweight SQL tasks on smaller workers. See Configure worker queues for setup instructions.

• Kubernetes executor: Each task runs in its own dedicated Kubernetes Pod rather than in a shared worker. You configure Pod resources through default Pod size settings and per-task pod_override configurations. See Configure Kubernetes Pod resources and Customize a task’s Kubernetes Pod.

For details on worker autoscaling behavior, see Configure the Celery executor or Configure the Astro executor.

Scheduler

The Airflow scheduler is responsible for monitoring task execution and triggering downstream tasks when the dependencies are met.

Scheduler resources must be set for each Deployment and are managed separately from cluster-level infrastructure. To ensure that your tasks have the CPU and memory required to complete successfully on Astro, you can provision the scheduler with varying amounts of CPU and memory.

The scheduler always receives a fixed allocation of 1 vCPU and 2 GiB memory in remote execution mode. You cannot change scheduler resources for remote execution Deployments. You can enable High Availability to run two schedulers in this mode, each with the same fixed resources.

Unlike workers, schedulers do not autoscale. The resources you set for them are the resources you have regardless of usage. For more information about how scheduler configuration affects resources usage, see Pricing.

Astronomer Deployments run a single scheduler by default. You can configure your scheduler to have different amounts of resources based on how many tasks you need to schedule for hosted execution Deployments. For remote execution Deployments, scheduler resources are fixed and not configurable.

You can also enable High Availability to run two instances of PGBouncer and the Airflow Scheduler.

Size options

Astro separates the scheduler and Dag processor for some Deployment sizes, which improves security and reliability because the SchedulerJob and DAGProcessorJob can run on separate Pods. Consider the following details when choosing your scheduler and Dag processor sizes:

• To use the separate scheduler and Dag processor, you must use at least version 9.7.0 of Astro Runtime. If your Deployment uses a lower Runtime version, then the scheduler and Dag processor run on the same Pod, and the Extra Large Deployment size is not available.
• For Small Deployments, the scheduler and Dag processor run on the same Pod.
• Extra Large Deployments have two Dag processors allocated per Deployment.

Astronomer recommends using

Medium

or larger for production Deployments for improved reliability and performance. This aligns with Apache Airflow best practices of running the scheduler and Dag processor as separate processes. For Airflow 3.x Deployments,

Medium

or larger is highly recommended, as open source Airflow 3 has fully separated the scheduler and Dag processor into distinct processes.

The following table lists all possible scheduler and Dag processor sizes for Astro Hosted:

Size	vCPU	Memory	Ephemeral Storage
Small (Up to ~50 dags)	1	2GiB	5 GiB
Medium (Up to ~250 dags)	Scheduler: 1 Dag processor: 1	Scheduler: 2 GiB Dag processor: 2 GiB	Scheduler: 1 GiB Dag processor: 4 GiB
Large (Up to ~1000 dags)	Scheduler: 1 Dag processor: 3	Scheduler: 2 GiB Dag processor: 6 GiB	Scheduler: 1 GiB Dag processor: 4 GiB
Extra Large (Up to ~2000 dags)	Scheduler: 1 Dag processor (x2): 3.5	Scheduler: 4 GiB Dag processor (x2): 6 GiB	Scheduler: 1 GiB Dag processor (x2): 4 GiB

Update scheduler size

1. In the Astro UI, select a Workspace, click Deployments, and then select a Deployment.

2. Click the Options menu of the Deployment you want to update, and select Edit Deployment.

   

3. In the Advanced section, choose a scheduler size. See Scheduler.

4. Click Update Deployment.

   The Airflow components of your Deployment automatically restart to apply the updated resource allocations. This action is equivalent to deploying code and triggers a rebuild of your Deployment image. If you’re using the Celery executor, currently running tasks have 24 hours to complete before their running workers are terminated. See What happens during a code deploy.

Triggerer

The Airflow triggerer executes deferred tasks in an asyncio event loop. Running a triggerer is essential for using deferrable operators.

One replica of the Airflow triggerer with 0.5 vCPU and 1.875 GiB is automatically included in your Deployment, and is the same size for all Deployments. The included triggerer can typically support up to 1000 concurrent running triggers with low to medium memory requirements. If your use case requires over 1000 concurrent running triggers and/or medium to heavy memory requirements, Astro supports customizing the count and size of the triggerer. Leverage the triggerer metrics available in Deployment Analytics to right-size the triggerer. Additional requested compute beyond the default specs will be charged at the A5 worker unit rate, similar to KE/KPO chargeback.

The following table lists all available environment variables for customizing the count and size of the triggerer.

Environment Variable	Default	Max
ASTRO_TRIGGERER_REPLICAS	1	8
ASTRO_TRIGGERER_RESOURCES_CPU	500m (0.5 vCPU)	5000m (5vCPU)
ASTRO_TRIGGERER_RESOURCES_MEMORY	1920Mi (1.875GiB)	15360Mi (15GiB)

Enable high availability

This is feature is only available if you are on the

Team

tier or above. See

Astro Plans and Pricing

.

By default, the Pods running your Deployment’s Airflow components are distributed across multiple nodes. When you enable high availability, Astro re-configures the Deployment to be more resilient and avoid a single point of failure by running replicas of key infrastructure components across different availability zones. Changes in high availability mode include:

• Running two schedulers so that at least one is always available
• Running two Dag processors in Deployments with a standalone Dag processor
• Running two webservers
• Running the following components on different nodes with preferred zone-level anti-affinity:
  • Scheduler
  • Dag processor
  • PG Bouncer
  • Webserver

This ensures that your dags can continue to run if there’s an issue with one of your Airflow components in a specific node or availability zone.

High availability mode is recommended for production Deployments to increase reliability and availability of scheduling, dag processing and the UI/API, while also providing protection against zonal outages.

Because this setting results in more resource usage, it increases the cost of your Deployment. See Pricing.

1. In the Astro UI, select a Workspace, click Deployments, and then select a Deployment.

2. Click the Options menu of the Deployment you want to update, and select Edit Deployment.

   

3. In the Basic section, click the toggle to On for High Availability.

4. Select Update Deployment to save your changes.

Enable development mode

Enabling Development Mode on a Deployment allows you to safely separate development and production environments on Astro. Deployments in Development Mode can be hibernated to control costs. In addition to marking a Deployment as Development Mode, you can also control development and testing environments by creating ephemeral preview Deployments. To learn more about which approach is right for you, see Manage development Deployments on Astro.

Development Mode is a prerequisite to hibernate a Deployment and must be enabled at Deployment creation. It cannot be turned on after a Deployment has been created, but it can be turned off after a Deployment is created.

Development Deployments have the following constraints:

• The Small Scheduler (1 vCPU, 2 GiB RAM) is the only scheduler size supported.
• Development Deployments cannot have a P1 ticket raised against them.
• Development Deployments will not be treated as “production” by support.

Hibernate a development Deployment

Public Preview

This feature is in

Public Preview

.

When you create a Deployment on Astro, you pay for the infrastructure resources that are required to run the Deployment for the duration that it’s active. In development environments when you aren’t always running tasks, you can hibernate, or scale down, all Deployment resources on a specified schedule. When you hibernate a Deployment, all Deployment configurations are preserved, but computing resources are scaled to zero.

For example, if you only need to test a dag during working hours, you can set a hibernation schedule for 5:00 PM until 9:00 AM on Monday through Friday. During this time, your Deployment settings are preserved and your cost on Astro for the Deployment is $0. When the hibernation schedule ends, you can resume using the Deployment. Waking up a Deployment from hibernation is faster than creating a new Deployment and preserves all of your configurations.

Use Deployment hibernation to ensure that:

• You only pay for the resources that you need, when you need them.
• You don’t have to delete a Deployment in order to avoid the cost of the Deployment.
• You don’t have to recreate development environments and re-enter Deployment configurations.

Prerequisites

You can hibernate a Deployment only if you enabled Development Mode when you created the Deployment. Deployments without this setting enabled can’t be hibernated.

Create a wake schedule

Before you create or edit a wake schedule for a Deployment, consider the following constraints:

• The Deployment must have the Development Mode setting turned on. This setting can be turned on only when you create a Deployment.
• The High Availability feature is not supported. A Deployment with a hibernation schedule cannot be highly available.
• Deployments with hibernation schedules are not required to meet the uptime SLAs of standard production Deployments.

The following instructions show you how to configure wake schedules in the Astro UI. You can also configure wake schedules programmatically via the Astro API.

To manage wake schedules:

1. In the Astro UI, select a Workspace, click Deployments, then select a Deployment.
2. Click Details. In the Advanced section of your Deployment configuration, click Edit.
3. In the Wake Schedules section, set one or more schedules using either the calendar-based interface or cron expressions. You can switch between the two interfaces by clicking the Switch to cron scheduling or Switch to simple scheduling buttons.
   To keep the Deployment hibernated indefinitely, select the Never Wake schedule type in the simple scheduling interface. You can only enable one Never Wake schedule at a time. You can wake up the Deployment by clicking Wake Up Deployment in the Deployment’s More actions menu, or programmatically via the Astro API.
4. Click Update Deployment.

Astro sets all cron schedules for hibernation in UTC. If you’re running a Deployment in another time zone, you must convert the cron expression for your time zone to UTC. For example, if you want your Deployment to hibernate at 17:00 EST, you use

0 22 * * *

(22:00 UTC) as the cron expression.

When your hibernation schedule starts:

• Your Deployment shows a Hibernating status in the Astro UI:

  

• Any task that was previously running will be killed and marked as failed.

• Tasks and dags do not run. Task instances that were already running or scheduled at the time of hibernation will fail and trigger any related notifications.

• No Deployment resources are available. This includes the scheduler, webserver, and all workers.

• You can’t access the Airflow UI for the Deployment.

• You can’t deploy project images or dags to the Deployment.

When your hibernation schedule ends, the Deployment will start any dag runs for data intervals that were missed during hibernation for dags with catchup=True. To avoid incurring additional resource costs, Astronomer recommends disabling catchup on dags in hibernating Deployments.

Manually hibernate a Deployment

Instead of creating a regular hibernation schedule, you can manually hibernate a development Deployment from the Astro UI. This is recommended if you’re not sure when you’ll need to use the Deployment again after hibernating it.

1. In the Astro UI, select a Workspace, click Deployments, and then select a Deployment.
2. Click the More Actions menu of the Deployment you want to update, then select Hibernate Deployment.
3. Configure the manual hibernation period, then click Confirm.

If you need to run a task or dag on a Deployment that is currently in hibernation, you can manually wake up a Deployment from hibernation before the end of its schedule.

1. In the Astro UI, select a Workspace, click Deployments, and then select a Deployment.

2. Click the More Actions menu of the Deployment you want to update, then select Wake Up Deployment.

3. Select one of the following options for how you want your Deployment to wake up:

   • Wake until further notice: Your Deployment wakes up immediately for an indefinite period and ignores any configured hibernation schedules.
   • Wake until set time and date: Specify a time that the Deployment should go back into hibernation after waking up.
   • Remove override and return to normal schedule: The Deployment returns to following your configured hibernation schedules.

4. Click Confirm.