Configure AWS PrivateLink for Remote Execution Agents

Configure private connectivity between Remote Execution Agents and the Astro orchestration plane using AWS PrivateLink.

Airflow 3

This feature is only available for Airflow 3.x Deployments.

AWS PrivateLink enables private connectivity between your Remote Execution Agents and the Astro orchestration plane without exposing traffic to the public internet. This guide explains how to set up a VPC Endpoint in your AWS environment to establish secure communication with Astro.

Overview

By default, Remote Execution Agents communicate with the Astro orchestration plane over the public internet. With AWS PrivateLink, you can route this traffic through a private connection within AWS, which provides enhanced security and can simplify network configurations for organizations with strict security requirements.

The setup involves creating a VPC Endpoint in your AWS account that connects to Astronomer’s VPC Endpoint Service. Once configured, your Remote Execution Agents can communicate with Astro through this private connection.

Prerequisites

• An Astro Deployment configured for Remote Execution.
• Remote Execution Agents installed in an AWS environment.
• Access to the AWS Console with permissions to create VPC Endpoints and modify Route53 configurations.

Astro-side configuration

Before you can create a VPC Endpoint, Astronomer must configure the VPC Endpoint Service on the Astro side. Contact Astronomer Support with the following information:

• Your Astro Cluster ID.
• The AWS Account ID where your Remote Execution Agents are running.
• The AWS Region where your Remote Execution Agents are running.

Astronomer Support will provide you with the VPC Endpoint Service name, Service region, and supported Availability Zones required to create your VPC Endpoint.

If your Remote Execution Agents run in a different AWS region than the Astro orchestration plane, inform Astronomer Support. Additional configuration may be required on the Astro side, such as adding your region to the VPC Endpoint Service cross-region configuration or adding your AWS account to the allowed principals list.

Create a VPC Endpoint

After receiving the VPC Endpoint Service name from Astronomer Support, create a VPC Endpoint in your AWS account.

1. Navigate to VPC Endpoints
2. In the AWS Console, go to VPC > Endpoints.
3. Start the endpoint creation wizard
4. Click Create endpoint to begin the configuration.
5. Configure the endpoint
6. Set the following values:

Subnet selection shows your subnets in the Availability zones supported by the VPCe Service. If there is a mismatch, you must create subnet(s) in the zones provided by Astronomer Support.

Configure DNS resolution

After creating the VPC Endpoint, configure DNS so that your Remote Execution Agents resolve the Astro orchestration plane hostname to the private endpoint IP addresses.

Configure Route53 private hosted zone

1. Create a private hosted zone
3. Create an alias record

Verify the connection

After completing the configuration, verify that your Remote Execution Agents can communicate with Astro through the private endpoint. Validate in the Astro UI that the agents are heart beating and reporting a Healthy status. You can also verify from within your network using the below instructions.

1. Connect to a host within your VPC that has network access to the VPC Endpoint.
2. Run a DNS lookup to confirm the hostname resolves to a private IP address:

nslookup <AstroClusterId>.external.astronomer.run

The response should show the private IP addresses assigned to your VPC Endpoint rather than public IP addresses.

3. Test connectivity to the endpoint:

curl -v https://<AstroClusterId>.external.astronomer.run

The expected response is 404 page not found. If the connection is successful, your Remote Execution Agents will use the private endpoint for all communication with the Astro orchestration plane.

Multiple Remote Execution Agents

If you have multiple Remote Execution Agents across different VPCs, you can either create a VPC Endpoint in each VPC, or use a single VPC Endpoint and configure network routing between VPCs.

The following table summarizes the actions required based on your configuration:

Configuration	Yes	No
Same AWS region	No additional actions	Attach VPC to the Route53 private hosted zone
Different AWS region	No additional actions	Contact Astronomer Support
Different AWS account	No additional actions	Contact Astronomer Support

If you previously created a Route53 private hosted zone, you can associate additional VPCs with the same hosted zone rather than creating new zones for each VPC.

Restrict traffic to the private endpoint

After verifying that the private endpoint works correctly, you can optionally configure your Remote Execution Agents to only allow traffic through the VPC Endpoint. This ensures that all communication with Astro uses the private connection.

To restrict traffic:

1. Take note of your Astro Cluster ID, under Organization Settings > Clusters > Cluster details.
2. In the Astro UI, navigate to your Deployment and go to Settings.
3. In your Deployment Advanced settings, add the cluster CIDR range to the Allowed IP address ranges list.

This configuration ensures that only traffic coming through the VPC Endpoint Service can reach the Deployment.

Troubleshooting

VPC Endpoint shows “pending acceptance”

The VPC Endpoint Service may require manual acceptance of endpoint connections, if still in pending state after 5 minutes. Contact Astronomer Support to approve your endpoint connection request.

DNS resolution returns public IP addresses

Verify that your Route53 private hosted zone is correctly configured and associated with the VPC where you are testing.

Connection timeouts

Check that the security group attached to the VPC Endpoint allows inbound traffic on port 443 from the subnets where your Remote Execution Agents are running.