Configure Azure Private Link for Remote Execution Agents

Configure private connectivity between Remote Execution Agents and the Astro orchestration plane using Azure Private Link.

Airflow 3

This feature is only available for Airflow 3.x Deployments.

Azure Private Link enables private connectivity between your Remote Execution Agents and the Astro orchestration plane without exposing traffic to the public internet. This guide explains how to set up a Private Endpoint in your Azure environment to establish secure communication with Astro.

Overview

By default, Remote Execution Agents communicate with the Astro orchestration plane over the public internet. With Azure Private Link, you can route this traffic through a private connection within Azure, which provides enhanced security and can simplify network configurations for organizations with strict security requirements.

The setup involves creating a Private Endpoint in your Azure subscription that connects to Astronomer’s Private Link Service. Once configured, your Remote Execution Agents can communicate with Astro through this private connection.

Prerequisites

• An Astro Deployment configured for Remote Execution.
• Remote Execution Agents installed in an Azure environment (Azure Kubernetes Service (AKS)).
• Access to the Azure portal with permissions to create Private Endpoints, Private DNS zones, and modify networking configurations.

Astro-side configuration

Before you can create a Private Endpoint, Astronomer must configure the Private Link Service on the Astro side. Contact Astronomer support with the following information:

• Your Astro Cluster ID.

Astronomer support will provide you with the Private Link Service alias required to create your Private Endpoint.

Create a Private Endpoint

After receiving the Private Link Service alias from Astronomer Support, create a Private Endpoint in your Azure subscription.

1. Gather required information
2. Before starting, collect the following:
4. Navigate to Private Endpoints
5. In the Azure portal, go to Private Link Overview > Private endpoints.
6. Start the endpoint creation wizard
7. Click Create to begin the configuration.
8. Configure the endpoint
9. Set the following values:
11. Configure the resource connection
12. Set the following values:
14. Configure networking
15. Set the following values:
17. Create the endpoint
18. Review your configuration and click Create.

After creating the Private Endpoint, contact Astronomer support to approve your endpoint connection request. You can proceed with the DNS configuration while waiting for approval.

Configure DNS resolution

After creating the Private Endpoint, configure DNS so that your Remote Execution Agents resolve the Astro orchestration plane hostname to the Private Endpoint IP address.

Create a private DNS zone

1. Navigate to Private DNS zones
2. In the Azure portal, go to Private DNS zones.
3. Create a new zone
5. Link the DNS zone to your VNet
7. Create an A record

Verify the connection

After Astronomer support approves your endpoint connection, verify that your Remote Execution Agents can communicate with Astro through the Private Endpoint. Validate in the Astro UI that the agents are heartbeating and reporting a Healthy status. You can also verify from within your network using the following instructions.

1. Connect to a host within your VNet that has network access to the Private Endpoint.
2. Run a DNS lookup to confirm the hostname resolves to a private IP address:

nslookup <AstroClusterId>.external.astronomer.run

The response should show the private IP address assigned to your Private Endpoint rather than a public IP address.

3. Test connectivity to the endpoint:

curl -v https://<AstroClusterId>.external.astronomer.run

The expected response is 404 page not found. If the connection is successful, your Remote Execution Agents will use the Private Endpoint for all communication with the Astro orchestration plane.

Multiple Remote Execution Agents

Only one Private Link Service is required per Astro cluster. If you have multiple Remote Execution Agents across different VNets, you can either create a Private Endpoint in each VNet (the Private Link Service alias remains unchanged) or use a single Private Endpoint across your network.

If you previously created a private DNS zone, you can associate additional VNets with the same zone rather than creating new zones for each VNet.

Restrict traffic to the Private Endpoint

After verifying that the Private Endpoint works correctly, you can optionally configure your Remote Execution Agents to only allow traffic through the Private Endpoint. This ensures that all communication with Astro uses the private connection.

To restrict traffic:

1. Take note of your Astro Cluster ID, under Organization Settings > Clusters > Cluster details.
2. In the Astro UI, navigate to your Deployment and go to Settings.
3. In your Deployment Advanced settings, add the cluster CIDR range to the Allowed IP address ranges list.

This configuration ensures that only traffic coming through the Private Link Service can reach the Deployment.

Troubleshooting

Private Endpoint shows “pending” connection state

The Private Link Service requires manual acceptance of endpoint connections. Contact Astronomer support to approve your endpoint connection request.

DNS resolution returns public IP addresses

Verify that your private DNS zone is correctly configured and linked to the VNet where you are testing. Ensure the A record points to the correct Private Endpoint IP address.

Connection timeouts

Check that your network security group (NSG) rules allow outbound traffic on port 443 from the subnets where your Remote Execution Agents are running to the Private Endpoint.