Skip to main content

Documentation Index

Fetch the complete documentation index at: https://astronomer-preview.mintlify.app/llms.txt

Use this file to discover all available pages before exploring further.

This reference describes configuration values for the Remote Execution Agent Helm chart. For complete configuration options, see the values.yaml file downloaded from the Astro UI.

​
Required configuration values

The following values must be configured before installing the Helm chart:

​
Agent authentication

agentToken / agentTokenSecretName / agentTokenFile You must specify exactly one of these to provide the agent token generated in the Astro UI.
  • agentToken: Token value as plain text in values.yaml (not recommended for production)
  • agentTokenSecretName: Name of existing Kubernetes secret containing the token
  • agentTokenFile: Path to file containing the token (agent reads at runtime)
See Agent token configuration for detailed instructions.

​
Image registry access

imagePullSecretName / imagePullSecretData You must specify exactly one of these to allow agents to pull images from the registry.
  • imagePullSecretName: Name of existing Kubernetes secret with Docker credentials
  • imagePullSecretData: Docker config JSON as string (Helm creates secret named image-pull-secret)
See Image pull secret configuration for detailed instructions.

​
Kubernetes namespace

namespace Kubernetes namespace where the agent will be deployed.
  • If createNamespace: true, Helm creates the namespace
  • If createNamespace: false, namespace must exist before installation
If using agentTokenSecretName and imagePullSecretName, set createNamespace: false and create the namespace manually with secrets already present.
See Install in restricted Kubernetes namespace for restricted namespace configuration.

​
Resource name prefix

resourceNamePrefix Name prefix for all Kubernetes resources (Deployments, ConfigMaps, Secrets) created by the Helm chart.

​
Secrets backend

secretBackend Airflow secrets backend class for accessing connections and variables. Required for agent operation. Supported backends:
  • airflow.providers.amazon.aws.secrets.secrets_manager.SecretsManagerBackend
  • airflow.providers.microsoft.azure.secrets.key_vault.AzureKeyVaultBackend
  • airflow.providers.google.cloud.secrets.secret_manager.CloudSecretManagerBackend
  • airflow.providers.hashicorp.secrets.vault.VaultBackend
  • airflow.secrets.local_filesystem.LocalFilesystemBackend (not recommended for production)
See Configure secrets backend for detailed configuration instructions.

​
XCom backend

xcomBackend Airflow XCom backend class for passing data between tasks. Required for agent operation. Typically set to: airflow.providers.common.io.xcom.backend.XComObjectStorageBackend See Configure XCom backend for detailed configuration instructions.

​
DAG bundles

dagBundleConfigList JSON string defining how agents access dag code. Required for running dags. See Configure DAG sources for detailed configuration instructions.

​
Common environment variables

commonEnv Environment variables applied to all agent components (worker, DAG processor, triggerer). Used to configure secrets backend parameters, XCom paths, logging settings, and other Airflow configuration. Example: 
commonEnv:
  - name: AIRFLOW__SECRETS__BACKEND_KWARGS
    value: '{"connections_prefix": "airflow/connections", "variables_prefix": "airflow/variables"}'
  - name: AIRFLOW__COMMON_IO__XCOM_OBJECTSTORAGE_PATH
    value: "s3://bucket/xcom"

​
Worker resource configuration

workers Workers are configured as a list in values.yaml. Each entry defines a worker Deployment with its own name, resource allocation, replica count, and optional queue assignment.
ParameterDescriptionDefault
nameUnique identifier for the worker Deployment. Used in Kubernetes resource names.default-worker
replicasNumber of worker Pod replicas. Ignored when hpa.enabled is true.1
queuesComma-separated list of Airflow queues this worker listens on.default
resources.requests.cpuMinimum CPU allocated to the worker Pod.—
resources.requests.memoryMinimum memory allocated to the worker Pod.—
resources.limits.cpuMaximum CPU the worker Pod can use.—
resources.limits.memoryMaximum memory the worker Pod can use.—
envList of environment variables specific to this worker.[]
volumesAdditional volumes to mount on the worker Pod.[]
volumeMountsMount paths for the additional volumes.[]
nodeSelectorKubernetes node selector for scheduling worker Pods.{}
tolerationsKubernetes tolerations for scheduling worker Pods.[]
serviceAccount.nameCustom service account name. Overrides the default {{ resourceNamePrefix }}-worker-{{ worker.name }}.—
serviceAccount.createWhether the Helm chart creates the service account. Set to false when using a pre-existing service account.true
terminationGracePeriodSecondsThe grace period for the worker Pod to finish existing tasks before terminating.600
Example with two workers: 
workers:
  - name: default-worker
    replicas: 2
    queues: "default"
    resources:
      requests:
        cpu: "500m"
        memory: "1Gi"
      limits:
        cpu: "2"
        memory: "4Gi"
  - name: high-memory-worker
    replicas: 1
    queues: "high-memory"
    resources:
      requests:
        cpu: "1"
        memory: "4Gi"
      limits:
        cpu: "4"
        memory: "16Gi"
When you configure multiple workers, each worker creates a separate Kubernetes Deployment. The service account name for each worker defaults to {{ resourceNamePrefix }}-worker-{{ worker.name }}. If you use IRSA (AWS), Workload Identity (GCP), or managed identity (Azure), annotate each worker’s service account.

​
Horizontal Pod Autoscaler

workers[].hpa Each worker supports a Horizontal Pod Autoscaler (HPA) configuration to automatically scale the number of worker Pod replicas based on resource utilization or custom metrics. When hpa.enabled is true, the Helm chart creates a HorizontalPodAutoscaler resource for the worker Deployment. The replicas value is ignored because the HPA controls replica count.
ParameterDescriptionDefault
hpa.enabledEnable the Horizontal Pod Autoscaler for this worker.false
hpa.minReplicasMinimum number of worker Pod replicas.1
hpa.maxReplicasMaximum number of worker Pod replicas.10
hpa.metricsList of metric targets that the HPA uses to make scaling decisions. Follows the Kubernetes HPA metrics spec.—
Example with CPU-based autoscaling: 
workers:
  - name: default-worker
    queues: "default"
    resources:
      requests:
        cpu: "500m"
        memory: "1Gi"
      limits:
        cpu: "2"
        memory: "4Gi"
    hpa:
      enabled: true
      minReplicas: 1
      maxReplicas: 5
      metrics:
        - type: Resource
          resource:
            name: cpu
            target:
              type: Utilization
              averageUtilization: 80
You must set resources.requests for the metrics you use in HPA targets. For example, CPU-based autoscaling requires resources.requests.cpu to be set. Without resource requests, the HPA cannot calculate utilization percentages.
You can combine multiple metrics to define more sophisticated scaling behavior. The HPA evaluates all specified metrics and scales to the highest recommended replica count. Example with CPU and memory metrics: 
    hpa:
      enabled: true
      minReplicas: 2
      maxReplicas: 10
      metrics:
        - type: Resource
          resource:
            name: cpu
            target:
              type: Utilization
              averageUtilization: 75
        - type: Resource
          resource:
            name: memory
            target:
              type: Utilization
              averageUtilization: 80

​
Triggerer resource configuration

triggerer The triggerer runs deferred tasks asynchronously. Configure the triggerer to control replica count, async capacity, resource allocation, and Pod-level settings.
ParameterDescriptionDefault
replicasNumber of triggerer Pod replicas.1
asyncSlotsNumber of concurrent triggers the triggerer Pod can compute.1000
imageDocker image for the triggerer. Defaults to the top-level image if not set.—
imagePullPolicyImage pull policy for the triggerer. Defaults to the top-level imagePullPolicy if not set.—
resources.limits.cpuMaximum CPU the triggerer Pod can use.1
resources.limits.ephemeral-storageMaximum ephemeral storage the triggerer Pod can use.1Gi
resources.limits.memoryMaximum memory the triggerer Pod can use.2Gi
resources.requests.cpuMinimum CPU allocated to the triggerer Pod.1
resources.requests.ephemeral-storageMinimum ephemeral storage allocated to the triggerer Pod.1Gi
resources.requests.memoryMinimum memory allocated to the triggerer Pod.2Gi
envList of environment variables specific to the triggerer.[]
livenessProbeLiveness probe configuration for the triggerer Pod.See following example
readinessProbeReadiness probe configuration for the triggerer Pod.See following example
podSecurityContextPod security context for the triggerer Pod. By default, the agent runs as a non-root user with UID 50000 and group ID 50000.~
containerSecurityContextSecurity context for the triggerer container.{}
initContainersInit containers to add to the triggerer Pod.[]
extraContainersSidecar containers to add to the triggerer Pod.[]
volumesAdditional volumes to mount on the triggerer Pod.[]
volumeMountsMount paths for the additional volumes.[]
nodeSelectorKubernetes node selector for scheduling triggerer Pods.~
affinityAffinity rules for the triggerer Pod.{}
tolerationsKubernetes tolerations for scheduling triggerer Pods.[]
Example: 
triggerer:
  replicas: 2
  asyncSlots: 500
  resources:
    limits:
      cpu: "1"
      ephemeral-storage: "1Gi"
      memory: "2Gi"
    requests:
      cpu: "1"
      ephemeral-storage: "1Gi"
      memory: "2Gi"
  livenessProbe:
    httpGet:
      path: healthz
      port: 39091
    initialDelaySeconds: 30
    periodSeconds: 5
    failureThreshold: 3
    successThreshold: 1
    timeoutSeconds: 5
  readinessProbe:
    httpGet:
      path: healthz
      port: 39091
    initialDelaySeconds: 30
    periodSeconds: 5
    failureThreshold: 3
    successThreshold: 1
    timeoutSeconds: 5
  podSecurityContext:
    runAsUser: 50000
    fsGroup: 50000
    runAsNonRoot: true
    seccompProfile:
      type: RuntimeDefault
  containerSecurityContext:
    allowPrivilegeEscalation: false
    readOnlyRootFilesystem: true
    capabilities:
      drop:
        - ALL
  initContainers:
    - name: vault-agent
      image: vault:1.13.0
      command: ["vault", "agent", "-config=/etc/vault/config.hcl"]
      volumeMounts:
        - name: vault-config
          mountPath: /etc/vault
      env:
        - name: VAULT_ADDR
          value: "https://vault.example.com"
        - name: VAULT_TOKEN
          valueFrom:
            secretKeyRef:
              key: token
              name: vault-token
  extraContainers:
    - name: logging-sidecar
      image: timberio/vector:0.45.0-debian
      env:
        - name: VECTOR_CONFIG
          value: /etc/vector/vector.yaml
      volumeMounts:
        - name: vector-config
          mountPath: /etc/vector
      resources:
        limits:
          cpu: "0.5"
          memory: "256Mi"
        requests:
          cpu: "0.5"
          memory: "256Mi"
  volumes:
    - name: task-logs
      emptyDir: {}
    - name: vector-config
      configMap:
        name: vector-config
  volumeMounts:
    - name: task-logs
      mountPath: /var/log/airflow
      readOnly: true
  nodeSelector:
    node.kubernetes.io/instance-type: c5.large
To run multiple triggerers, increase replicas. The configured number of replicas runs continuously. For restricted namespaces with Pod security standards set to restricted, configure podSecurityContext and containerSecurityContext to meet your cluster’s requirements.

​
Optional configuration

​
Logging sidecar

loggingSidecar Optional sidecar for exporting task logs to external platforms or viewing logs in the Airflow UI before task completion. See Configure logging sidecar for configuration instructions.

​
OpenLineage

openLineage Optional configuration for data lineage collection.
You must configure OpenLineage to useAstro Observewith Remote Execution Deployments.
See Configure OpenLineage for configuration instructions.

​
Sentinel monitoring

sentinel Monitoring service for agent health reporting (agent version 1.2.0+). Astronomer recommends enabling Sentinel for all deployments. See Sentinel for Remote Execution Agents for configuration instructions.

​
Cloud provider annotations

annotations and labels Kubernetes annotations and labels to configure Pods to run using a specific IAM role (AWS), workload identity (GCP) or managed identity (Azure).

​
Helm commands

After the Remote Execution Agent is installed, any updates to the agent use the helm upgrade command.

​
Install agent

helm repo add astronomer https://helm.astronomer.io
helm repo update
helm install astro-agent astronomer/astro-remote-execution-agent -f values.yaml

​
Update agent

helm upgrade astro-agent astronomer/astro-remote-execution-agent -f values.yaml

​
View current configuration

helm get values astro-agent