Skip to main content

Documentation Index

Fetch the complete documentation index at: https://astronomer-preview.mintlify.app/llms.txt

Use this file to discover all available pages before exploring further.

This guide provides everything you need to set up Remote Execution on Astro, including prerequisites, configuration requirements, and a step-by-step checklist.

​
What you’ll configure

Remote Execution requires both required and optional components. Understand what you need before starting setup.

​
Required components

You must configure these components for Remote Execution to function:
ComponentPurposeWhen configured
Remote Execution AgentsExecute Airflow tasks in your infrastructureRegister and configure agents
Secrets BackendSecurely store and access Airflow connections and variablesConfigure secrets backend
XCom BackendPass data between tasks using object storageConfigure XCom backend
Dag sourcesDefine how agents access Dag code (Git or local)Configure Dag sources
All required components must be configured before agents can successfully execute tasks. The setup checklist below provides the recommended order.
ComponentPurposeDocumentation
SentinelMonitor agent health and report status to the orchestration planeEnable Sentinel

​
Optional components

These components enhance functionality but aren’t required for basic operation:
ComponentPurposeDocumentation
LoggingExport task logs to external platforms or object storageConfigure logging
OpenLineageEnable data lineage and observability in Astro ObserveConfigure OpenLineage
AWS PrivateLinkEnable private connectivity between agents and AstroConfigure AWS PrivateLink
Azure Private LinkEnable private connectivity between agents and AstroConfigure Azure Private Link

​
Setup

Follow these steps in order to set up Remote Execution:

​
1. Meet prerequisites

Before you begin, ensure you have:
  • An Astro Deployment configured for Remote Execution mode. See Create a deployment.
  • Kubernetes 1.30 or later
Recommended Kubernetes configurationConfigure singleProcessOOMKill: true in your kubelet configuration. With this setting, Kubernetes kills only the process that runs out of memory instead of the entire Pod. Without it, one task’s out-of-memory error kills all tasks on the worker and loses all task logs.See Kubernetes documentation for details.
  • Helm 3 or later
  • Permissions to create Astro agents
  • A Deployment API token with Deployment Admin permissions
  • Network access from your Kubernetes cluster to Astro. Add https://<your-cluster-id>.external.astronomer.run/ to your organization’s allowlist. To find your cluster ID, go to Settings > Clusters in the Astro UI.

​
2. Register and configure agents

Install the Remote Execution Agent Helm chart in your Kubernetes cluster:
  1. Create an Agent Token in the Astro UI
  2. Download and configure the values.yaml file
  3. Install the Helm chart with your configuration
  4. Verify agent heartbeat in the Astro UI
See Register and configure agents.

​
3. Configure secrets backend

Configure a secrets backend so agents can access Airflow connections and variables:
  • AWS Secrets Manager
  • Azure Key Vault
  • Google Cloud Secret Manager
  • HashiCorp Vault
See Configure secrets backend.

​
4. Configure XCom backend

Configure object storage for passing data between tasks:
  • AWS S3
  • Azure Blob Storage
  • GCP Cloud Storage
See Configure XCom backend.

​
5. Configure Dag sources

Define how agents access your Dag code:
  • GitDagBundle (recommended): Dags in a Git repository with automatic versioning
  • LocalDagBundle: Dags in container images or persistent volumes
See Configure DAG sources.

​
6. Enable Sentinel

Enable Sentinel to monitor agent health and report status to the orchestration plane. Sentinel detects Pod and component issues and monitors essential integrations such as XCom and secrets backends. See Sentinel for Remote Execution Agents.

​
7. Optionally configure logging

Set up task log export to view logs in external platforms or the Airflow UI:
  • Export logs with a logging sidecar
  • Link Airflow UI to external logging platform
  • Show logs in Airflow UI from object storage
See Configure logging.

​
8. Deploy your project

Deploy your Airflow project to the Remote Execution deployment:
  1. Initialize a Remote Execution project with Astro CLI
  2. Build and push the server image (orchestration plane)
  3. Build and push client images (execution plane)
  4. Update agents to use the new client image
See Deploy a Remote Execution project.

​
9. Verify agents are running

Confirm agents are healthy and processing tasks:
  1. Check agent heartbeat status in the Astro UI
  2. View agent Pods in your Kubernetes cluster
  3. Trigger a test Dag run and verify task execution
  4. Check task logs to confirm logging configuration

​
Next steps

After completing setup: