Register and configure agents

Register and configure agents

Airflow 3

This feature is only available for Airflow 3.x Deployments.

Remote Execution Agents execute Airflow tasks in your Kubernetes infrastructure. This guide covers registering agents with your Astro Deployment and installing the Helm chart.

Prerequisites

• Astro Deployment configured for Remote Execution mode. See Create a Deployment.
• Kubernetes 1.30 or later
• Helm 3 or later
• Deployment API token with Deployment Admin role to pull the base Astro Remote Execution Agent Image

Step 1: Create agent token

The agent token authenticates your agent to the Astro orchestration plane. Create this token before installing the Helm chart.

Save the token value in a secure location immediately after creation. You cannot retrieve it again. The limit is 50 agent tokens per Deployment.

Astro UI

1. Navigate to Deployment
2. In the Astro UI, select a workspace, click Deployments, and select your Remote Execution Deployment.
3. Open tokens view
4. Select the Remote Agents tab and toggle to the Tokens view.
5. Create token
7. Save token
8. Copy the agent token and save it securely. You will use this token in the Helm chart configuration.

Astro API

1. Get bearer token
2. Retrieve your Deployment API token. This authenticates API requests.
3. Get organization ID
4. Make a GET request to the organizations endpoint:
5. curl https://api.astronomer.io/platform/v1beta1/organizations \
     -H "Authorization: Bearer <token>"
   

6. Locate the id field in the response.
7. Get Deployment ID
8. Make a GET request to the deployments endpoint using your organization ID:
9. curl https://api.astronomer.io/platform/v1beta1/organizations/<organizationId>/deployments \
     -H "Authorization: Bearer <token>"
   

10. Locate the id field for your Remote Execution Deployment in the response.
11. Create agent token
12. Make a POST request to create the token. Replace the organizationId, deploymentId, and your-API-Deployment-token placeholders with your own values. You can optionally set a description and expiration period for the token.This example sets a 30-day expiration:
13. curl -X POST https://api.astronomer.io/iam/v1beta1/organizations/<organizationId>/deployments/<deploymentId>/agent-tokens \
      -H "Authorization: Bearer <your-API-Deployment-token>" \
      -H "Content-Type: application/json" \
      -d '{
      "description": "Production agent token",
      "name": "prod-agent",
      "tokenExpiryPeriodInDays": 30
    }'
    

14. Save token
15. Copy the token field from the response and save it securely.

Step 2: Install Helm chart

Astronomer recommends pulling both the Remote Execution Agent image and the Sentinel image and storing them in your private registry. Sentinel provides advanced monitoring and reporting for Remote Execution Agents, starting from version 1.2.0. The Agent base images are minimal, so you might need to add packages for your pipelines to function properly. Use either an Organization API token with the Org Owner role or a Deployment API token with the Deployment Admin role to authenticate.

1. Download values file
3. Configure required values
4. Update the following values in values.yaml. All other values have working defaults.
5. You must configure these values before installing the Helm chart:
7. See the Helm chart comments and Helm chart configuration reference for descriptions of values.
8. Pull agent image for private registries
9. If self-hosting the image, log in to the image registry with your Deployment API token:
10. docker login images.astronomer.cloud -u cli -p <your-token>
    

11. Sentinel image available with 1.2.0 and later

    Starting with Remote Execution Agent 1.2.0, a Sentinel image is published alongside the agent images to provide monitoring for Remote Execution Agents. The Sentinel image must be pulled separately. Astronomer recommends enabling Sentinel for all deployments. To enable Sentinel, configure the service in your values.yaml file. See Sentinel for Remote Execution Agents.

12. After you log in, you can pull the Remote Execution Agent and Sentinel images directly. To find the latest version and image path, refer to the Remote Execution Agent release notes for all currently hosted images and Remote Execution Agent image reference for their full URLs. For example:
13. docker pull images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.1-3-python-3.12-astro-agent-1.2.0
    

14. docker pull images.astronomer.cloud/baseimages/astro-remote-execution-sentinel:1.2.0
    

15. Configure scope for registry proxies

    If you use JFrog Artifactory or a similar registry management tool to mirror or proxy images.astronomer.cloud, you need to configure specific include patterns instead of using the default **/* pattern.

    The Deployment API token has limited scope and cannot fetch manifests for all repositories. Configure your remote registry to include only these specific paths:

    • baseimages/astro-remote-execution-agent
    • baseimages/astro-remote-execution-sentinel

    Without these specific patterns, you might encounter 403 Forbidden errors when JFrog attempts to crawl all repositories in the registry.

16. Pull the Remote Execution Agent image, apply customizations that your dags require, and push it to your private registry. Then update the values.yaml file to reference your customized image.
17. Install Helm chart
18. You must configure secretBackend in your values.yaml before running the Helm install. The installation fails if secretBackend has no value. See Configure secrets backend.

19. Run the following commands to install the agent:
20. helm repo add astronomer https://helm.astronomer.io
    helm repo update
    helm install astro-agent astronomer/astro-remote-execution-agent -f values.yaml
    

Step 3: Optionally set allowed IP ranges

Restrict Deployment access to specific IP address ranges for additional security or network isolation between environments.

1. Open Deployment settings
2. In the Astro UI, click the options menu for your Deployment and select Edit.
3. Add IP ranges

Step 4: Verify agent heartbeat

Confirm the agent is connected and healthy.

1. Check agent status
2. In the Astro UI, go to the Remote Agents tab. A healthy agent shows:
4. You can also verify locally that all agent client deployment Pods are running with kubectl get pods -n <namespace>. For more in-depth validation, check pod logs for heartbeat activity.
5. To verify that your agents can communicate with your Astro Orchestration plane:
7. Temporarily remove any configured allowed IP ranges if the agent is not starting up and reporting Healthy. If connecting using a public connection, your network team may need to allowlist the Astro cluster’s public load balancer’s public IP addresses (step 2) for outbound access from your VPC.

8. Configure dag bundles
9. After verifying agent health, configure how agents access DAG code. See Configure DAG sources.
10. Run test dag
11. Trigger a test DAG run to verify the agent executes tasks successfully.
12. If you expect tasks to run longer than the default grace period of 10 minutes, update the terminationGracePeriodSeconds parameter for your workers in values.yaml. This ensures that worker Pods have enough time to finish existing tasks before terminating. See Worker resource configuration.

HTTP/HTTPS proxy server support

Starting with Remote Execution Agent 1.3.2, the agents support running behind an HTTP(S) proxy server. Configure proxy settings using the HTTP_PROXY, HTTPS_PROXY, and NO_PROXY environment variables.

For Remote Execution Agent versions earlier than 1.3.2, proxy servers are not supported. If your Kubernetes environment automatically adds a proxy configuration to Pods, the agents will fail to establish an outbound connection to the orchestration plane. You might see errors similar to these in worker logs:

• "exc_type":"ReadError","exc_value":"[Errno 104] Connection reset by peer"
• "exc_type":"HTTPStatusError","exc_value":"Client error '400 Bad Request' for url ...

Workaround: Remove the proxy configuration from the agent Pods, or upgrade to Agent 1.3.2 or later.

Agent token configuration

Provide the agent token using one of these methods:

agentToken

Store the token directly in values.yaml:

agentToken: "<your-token-value>"

Storing tokens directly in values files exposes them in version control. Use agentTokenSecretName or agentTokenFile for better security.

agentTokenSecretName

Reference an existing Kubernetes secret containing the token:

kubectl -n <namespace> create secret generic agent-token \
  --from-file=token=token.txt

In values.yaml:

agentTokenSecretName: "agent-token"

agentTokenFile

Mount a file containing the token. The agent reads the token at runtime:

agentTokenFile: "/path/to/token/file"

Image pull secret configuration

Configure image pull secrets to authenticate with your container registry. The configuration differs depending on whether you pull images directly from Astronomer’s registry or from a self-hosted registry.

The image pull secret requires an Astro API token, not an agent token. Use either an Organization API token with the Org Owner role or a Deployment API token with the Deployment Admin role. The agent token created in Step 1 authenticates the agent to the Astro orchestration plane and cannot be used for pulling images.

Astronomer registry

Use this configuration when pulling images directly from images.astronomer.cloud.

imagePullSecretName

Reference an existing Kubernetes secret in your namespace:

kubectl create secret docker-registry -n <namespace> <secretName> \
  --docker-server=images.astronomer.cloud \
  --docker-username=cli \
  --docker-password=<your-astro-api-token>

In values.yaml:

imagePullSecretName: "<secretName>"

imagePullSecretData

Alternatively, provide Docker config JSON directly. The Helm chart creates a secret named image-pull-secret:

imagePullSecretData: |
  {
    "auths": {
      "images.astronomer.cloud": {
        "auth": "<base64-encoded-credentials>",
        "email": "<email>"
      }
    }
  }

Self-hosted registry

Use this configuration when pulling images from a self-hosted registry, proxy, or mirror.

If you use a proxy or mirror for images.astronomer.cloud, you still need an Astro API token to authenticate with the upstream Astronomer registry. Configure this in your proxy settings. See Pull agent image for private registries.

imagePullSecretName

Reference an existing Kubernetes secret in your namespace:

kubectl create secret docker-registry -n <namespace> <secretName> \
  --docker-server=<your-registry-endpoint> \
  --docker-username=<your-registry-username> \
  --docker-password=<your-registry-password>

In values.yaml:

imagePullSecretName: "<secretName>"

imagePullSecretData

Alternatively, provide Docker config JSON directly. The Helm chart creates a secret named image-pull-secret:

imagePullSecretData: |
  {
    "auths": {
      "<your-registry-endpoint>": {
        "auth": "<base64-encoded-username:password>",
        "email": "<email>"
      }
    }
  }

Manage Remote Execution Agents

You can take the following actions on your registered Remote Execution Agents:

• Cordon: Cordoning a Remote Execution Agent marks it as unavailable for scheduling new tasks, while allowing it to continue running and complete any tasks already in progress.

This allows you to gracefully remove the Agent from service without interrupting current workloads. For example, you can cordon an Agent to delete or perform maintenance, such as an upgrade, on the Agent or underlying infrastructure.

A cordoned Agent will not receive new work, but it remains active until all running tasks have finished. Once ready to reintroduce the Agent to the task pool, it can be uncordoned to resume normal operation.

• Uncordon: Uncordoning a Remote Execution Agent re-enables it to receive new tasks and resume normal scheduling.

• Delete: Deletes the Remote Execution Agent from the Deployment.

Remote Execution Agent maintenance policy

Each Remote Execution Agent minor version is maintained for 6 months from the release month.

See Agent maintenance policy for more details about versioning, support, and upgrade recommendations.

Next steps

After registering agents, configure the required components:

• Configure secrets backend - Required for agent operation
• Configure XCom backend - Required for passing data between tasks
• Configure DAG sources - Required for accessing DAG code

Related documentation

• Remote Execution overview
• Remote Execution Agent failure scenarios
• Helm chart configuration reference
• Install agents in restricted Kubernetes namespaces