Configure XCom backend

Configure XCom backend

Remote Execution Agents require a custom XCom backend to pass data between tasks. This guide covers configuring AWS S3, Azure Blob Storage, or GCP Cloud Storage as your XCom backend.

Overview

XCom (cross-communication) enables data exchange between Airflow tasks. By default, Airflow stores XCom values in the metadata database. With Remote Execution, the metadata database resides in the Astro orchestration plane while agents execute tasks in your infrastructure.

A custom XCom backend stores XCom data in object storage accessible to all agents in your execution plane. This allows tasks running on different agents to share data through a common storage location.

XCom limitations with callbacks

In Airflow 3, XCom values are stored externally in storage backends. The database only holds the location of each XCom value, and retrieving the actual XCom data requires a call to the execution API with a valid task token.

Currently, callbacks do not receive a task token, so they cannot retrieve XCom values from external backends. As a result, using XCom in callbacks is not supported at this time when using Remote Execution with custom XCom backends.

XCom support in triggers

XCom triggerer support is enabled with Agent Client 1.5.0+. Triggers running on the agent can retrieve and push XCom values when using a compatible client version.

Prerequisites

• Remote Execution Agent installed and registered
• Object storage bucket in your cloud provider
• Workload identity configured for your Kubernetes cluster (IRSA for AWS, Managed Identity for Azure, Workload Identity for GCP)
• Required Python packages installed in your agent image (see cloud-specific sections below)

Configure XCom backend

All XCom backend configurations use the XComObjectStorageBackend class from the Common IO provider. Choose your cloud provider below for specific configuration instructions.

AWS S3

Prerequisites

The Remote Execution Agent image must include these Python packages:

• apache-airflow-providers-amazon
• apache-airflow-providers-common-io
• s3fs

Create S3 bucket

Create an S3 bucket for XCom storage with the following recommended settings:

• Versioning: Enable for data recovery
• Encryption: Enable server-side encryption (SSE-S3 or SSE-KMS)
• Lifecycle policy: Configure automatic deletion of old XCom objects

Configure IAM role

Create an IAM role with permissions to read and write to your XCom bucket. Attach this policy to the role:

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": [
        "s3:GetObject",
        "s3:PutObject",
        "s3:DeleteObject",
        "s3:ListBucket"
      ],
      "Resource": [
        "arn:aws:s3:::your-xcom-bucket",
        "arn:aws:s3:::your-xcom-bucket/*"
      ]
    }
  ]
}

Configure the trust relationship to allow your Kubernetes service accounts to assume the role. See AWS IRSA documentation for details.

Update Helm values

Update your agent’s values.yaml file with the XCom backend configuration:

xcomBackend: "airflow.providers.common.io.xcom.backend.XComObjectStorageBackend"

commonEnv:
  - name: AIRFLOW__COMMON_IO__XCOM_OBJECTSTORAGE_PATH
    value: "s3://<connection-id>@<bucket-name>/<path-to-xcom>"
  - name: AIRFLOW__COMMON_IO__XCOM_OBJECTSTORAGE_THRESHOLD
    value: "0"  # Always store XComs in object storage
  - name: AIRFLOW__COMMON_IO__XCOM_OBJECTSTORAGE_COMPRESSION
    value: "zip"

annotations:
  eks.amazonaws.com/role-arn: arn:aws:iam::<AWS_ACCOUNT_ID>:role/<IAM_ROLE_NAME>

Replace:

• <connection-id>: Airflow connection ID for S3 (use aws_default or create a custom connection)
• <bucket-name>: Your S3 bucket name
• <path-to-xcom>: Path prefix for XCom objects (e.g., xcom)
• <AWS_ACCOUNT_ID>: Your AWS account ID
• <IAM_ROLE_NAME>: IAM role name created above

Apply configuration

Update your Helm release:

helm upgrade astro-agent astronomer/astro-remote-execution-agent -f values.yaml

Azure Blob Storage

Prerequisites

The Remote Execution Agent image must include these Python packages:

• apache-airflow-providers-microsoft-azure
• apache-airflow-providers-common-io

Create storage account and container

Create an Azure Storage account and container for XCom storage:

1. Create a storage account with these recommended settings:
   • Performance: Standard
   • Replication: Locally-redundant storage (LRS) or higher
   • Secure transfer: Required
2. Create a container named xcom (or your preferred name)
3. Note the storage account name

Configure managed identity

Create a managed identity and grant it access to your storage account:

1. Create a managed identity in Azure
2. Assign the Storage Blob Data Contributor role to the managed identity on your storage account
3. Configure workload identity federation to allow your Kubernetes service accounts to use the managed identity

See Azure Workload Identity documentation for details.

Update Helm values

Update your agent’s values.yaml file with the XCom backend configuration:

xcomBackend: "airflow.providers.common.io.xcom.backend.XComObjectStorageBackend"

commonEnv:
  - name: AIRFLOW_CONN_WASB_XCOM
    value: '{"conn_type":"azure","login":"<storage-account-name>","extra":"{\"anon\": false, \"account_name\":\"<storage-account-name>\",\"managed_identity_client_id\":\"<managed-identity-client-id>\",\"workload_identity_tenant_id\":\"<tenant-id>\"}"}'
  - name: AIRFLOW__COMMON_IO__XCOM_OBJECTSTORAGE_PATH
    value: "abfs://wasb_xcom@<container-name>/<path-to-xcom>"
  - name: AIRFLOW__COMMON_IO__XCOM_OBJECTSTORAGE_THRESHOLD
    value: "0"  # Always store XComs in object storage
  - name: AIRFLOW__COMMON_IO__XCOM_OBJECTSTORAGE_COMPRESSION
    value: "zip"

labels:
  azure.workload.identity/use: "true"

annotations:
  azure.workload.identity/client-id: "<managed-identity-client-id>"

Replace:

• <storage-account-name>: Your Azure storage account name
• <managed-identity-client-id>: Client ID of your managed identity
• <tenant-id>: Your Azure tenant ID
• <container-name>: Container name (e.g., xcom)
• <path-to-xcom>: Path prefix for XCom objects

Apply configuration

Update your Helm release:

helm upgrade astro-agent astronomer/astro-remote-execution-agent -f values.yaml

GCP Cloud Storage

Prerequisites

The Remote Execution Agent image must include these Python packages:

• apache-airflow-providers-google
• apache-airflow-providers-common-io
• gcsfs

Create GCS bucket

Create a GCS bucket for XCom storage with the following recommended settings:

• Location type: Region or Multi-region based on your requirements
• Storage class: Standard
• Access control: Uniform (bucket-level permissions)
• Encryption: Google-managed or customer-managed encryption keys

Configure IAM service account

Create a GCP service account and grant it access to your GCS bucket:

1. Create a service account in your GCP project
2. Grant the service account the Storage Object Admin role (roles/storage.objectAdmin) on your GCS bucket
3. Configure Workload Identity to allow your Kubernetes service accounts to use the GCP service account

To link your Kubernetes service account (KSA) to your GCP service account (GSA):

gcloud iam service-accounts add-iam-policy-binding <your-service-account>@<project>.iam.gserviceaccount.com \
  --role roles/iam.workloadIdentityUser \
  --member "serviceAccount:<project>.svc.id.goog[<namespace>/<ksa-name>]"

See GKE Workload Identity documentation for details.

Update Helm values

Update your agent’s values.yaml file with the XCom backend configuration:

xcomBackend: "airflow.providers.common.io.xcom.backend.XComObjectStorageBackend"

commonEnv:
  - name: AIRFLOW__COMMON_IO__XCOM_OBJECTSTORAGE_PATH
    value: "gs://<bucket-name>/<path-to-xcom>"
  - name: AIRFLOW__COMMON_IO__XCOM_OBJECTSTORAGE_THRESHOLD
    value: "0"  # Always store XComs in object storage
  - name: AIRFLOW__COMMON_IO__XCOM_OBJECTSTORAGE_COMPRESSION
    value: "zip"

annotations:
  iam.gke.io/gcp-service-account: <your-service-account>@<project>.iam.gserviceaccount.com

Replace:

• <bucket-name>: Your GCS bucket name
• <path-to-xcom>: Path prefix for XCom objects (e.g., xcom)
• <your-service-account>: Your GCP service account name
• <project>: Your GCP project ID

Apply configuration

Update your Helm release:

helm upgrade astro-agent astronomer/astro-remote-execution-agent -f values.yaml

Configuration options

XCom path format

The AIRFLOW__COMMON_IO__XCOM_OBJECTSTORAGE_PATH parameter defines where XCom objects are stored:

<protocol>://<connection-id>@<bucket-or-container>/<path-prefix>

• protocol: s3:// for AWS, abfs:// for Azure, gs:// for GCP
• connection-id: Airflow connection ID for authentication
• bucket-or-container: Storage bucket or container name
• path-prefix: Optional path prefix for organizing XCom objects

XCom threshold

The AIRFLOW__COMMON_IO__XCOM_OBJECTSTORAGE_THRESHOLD parameter controls when XCom values are stored in object storage:

• 0: Always store in object storage (recommended for Remote Execution)
• >0: Store in object storage only if value size exceeds threshold in bytes

Set this to 0 to ensure all XCom values are accessible to remote agents.

Compression

The AIRFLOW__COMMON_IO__XCOM_OBJECTSTORAGE_COMPRESSION parameter controls compression:

• zip: Compress XCom values (recommended)
• None: No compression

Compression reduces storage costs and transfer time for large XCom values.

Verify configuration

After applying your configuration, verify XCom backend functionality:

1. Trigger a test DAG that uses XCom to pass data between tasks
2. Check that tasks complete successfully
3. Verify XCom objects appear in your storage bucket
4. Review agent logs for any XCom-related errors

Next steps

After configuring XCom backend:

• Configure secrets backend - Store Airflow connections and variables securely
• Configure dag sources - Define how agents access DAG code
• Configure logging - Export task logs to external platforms

Related documentation

• XCom backend strategies
• Remote Execution overview
• Helm chart configuration reference