Set up Azure Key Vault as your secrets backend

Set up Azure Key Vault as your secrets backend

This topic provides setup steps for configuring Azure Key Vault as a secrets backend on Astro.

If you use a different secrets backend tool or want to learn the general approach on how to integrate one, see Configure a Secrets Backend.

Prerequisites

• A Deployment.
• The Astro CLI.
• An Astro project.
• An existing Azure Key Vault linked to a resource group.
• Your Key Vault URL. To find this, go to your Key Vault overview page > Vault URI.
• (Remote Execution Only) Helm installed
• (Remote Execution Only) The values.yaml file from the Register Agents modal in your Deployments>Agents page.

If you do not already have Key Vault configured, read Microsoft Azure documentation.

Step 1: Register Astro as an app on Azure

Steps 1 and 2 are only required if you are using service principal (client secret) authentication. If you prefer to use managed identity authentication, skip to Step 3 and follow the Managed Identity tab instructions.

Follow the Microsoft Azure documentation to register a new application for Astro.

At a minimum, you need to add a secret that Astro can use to authenticate to Key Vault.

Note the value of the application’s client ID and secret for Step 3.

Step 2: Create an access policy

If you use a managed identity to authenticate to Key Vault, skip to

Step 3

. Ensure your managed identity has an access policy or Azure RBAC role that grants it access to your Key Vault secrets.

Follow the Microsoft documentation to create a new access policy for the application that you just registered. The settings you need to configure for your policy are:

• Configure from template: Select Key, Secret, & Certificate Management.
• Select principal: Select the name of the application that you registered in Step 1.

Step 3: Set up Key Vault locally

Astro

In your Astro project, add the following line to your requirements.txt file:

apache-airflow-providers-microsoft-azure

Add the following environment variables to your .env file. Choose the option that matches your authentication method:

Client secret authentication:

AIRFLOW__SECRETS__BACKEND=airflow.providers.microsoft.azure.secrets.key_vault.AzureKeyVaultBackend
AIRFLOW__SECRETS__BACKEND_KWARGS={"connections_prefix": "airflow-connections", "variables_prefix": "airflow-variables", "vault_url": "<your-vault-url>", "tenant_id": "<your-tenant-id>", "client_id": "<your-client-id>", "client_secret": "<your-client-secret>"}

For client secret authentication, find your client ID in Azure Portal at App Registration page > Application (Client) ID. To find your tenant ID, go to App Registration page > Directory (tenant) ID. To find your client secret, go to App Registration Page > Certificates and Secrets > Client Secrets > Value.

Managed identity authentication:

Before using managed identity authentication, you must configure your Deployment with a workload identity. See the Azure tab in Authorize a Deployment to cloud resources using workload identity to set up your managed identity and authorize it to your Deployment.

AIRFLOW__SECRETS__BACKEND=airflow.providers.microsoft.azure.secrets.key_vault.AzureKeyVaultBackend
AIRFLOW__SECRETS__BACKEND_KWARGS={"connections_prefix": "airflow-connections", "variables_prefix": "airflow-variables", "vault_url": "<your-vault-url>", "managed_identity_client_id": "<your-managed-identity-client-id>", "workload_identity_tenant_id": "<your-tenant-id>"}

Remote Execution

Add the Azure Key Vault Backend to your project by updating your values.yaml file. Choose an authentication method:

Service Principal

Add the following to your values.yaml file to set the secrets backend class to use the Vault provider and configure your secrets backend kwargs:

secretBackend: "airflow.providers.microsoft.azure.secrets.key_vault.AzureKeyVaultBackend"

commonEnv:
  - name: AIRFLOW__SECRETS__BACKEND_KWARGS
    value: '{"connections_prefix": "airflow-connections", "variables_prefix": "airflow-variables", "vault_url": "<your-vault-url>", "tenant_id": "<your-tenant-id>", "client_id": "<your-client-id>", "client_secret": "<your-client-secret>"}'

You need to run the Remote Execution Agent with your Azure credentials to fetch from your secrets manager.

Managed Identity

Managed identity authentication is more secure because it eliminates the need to manage client secrets.

Prerequisites

Before updating your Helm values, configure a managed identity with access to your Key Vault:

1. Create a user-assigned managed identity in Azure, or use an existing one.
2. Grant the managed identity access to your Key Vault. In your Key Vault, create an access policy with Get and List permissions for secrets, and select your managed identity as the principal (see Step 2).
3. Configure Azure AD Workload Identity to allow your Kubernetes service accounts to use the managed identity.

Update Helm values

Add the following to your values.yaml file:

secretBackend: "airflow.providers.microsoft.azure.secrets.key_vault.AzureKeyVaultBackend"

commonEnv:
  - name: AIRFLOW__SECRETS__BACKEND_KWARGS
    value: '{"connections_prefix": "airflow-connections", "variables_prefix": "airflow-variables", "vault_url": "<your-vault-url>", "managed_identity_client_id": "<your-managed-identity-client-id>", "workload_identity_tenant_id": "<your-tenant-id>"}'

labels:
  azure.workload.identity/use: "true"

annotations:
  azure.workload.identity/client-id: "<your-managed-identity-client-id>"

Replace:

• <your-vault-url>: Your Azure Key Vault URL
• <your-managed-identity-client-id>: Client ID of your managed identity
• <your-tenant-id>: Your Azure tenant ID

For managed identity authentication, find your managed identity client ID in Azure Portal at Managed Identities > your identity > Client ID. To find your tenant ID, go to Microsoft Entra ID > Overview > Tenant ID.

This configuration tells Airflow to look for variable information at the airflow/variables/* path in Azure Key Vault and connection information at the airflow/connections/* path. You can now run a dag locally to check that your variables are accessible using Variable.get("<your-variable-key>").

By default, this setup requires that you prefix any secret names in Key Vault with airflow-connections or airflow-variables. If you don’t want to use prefixes in your Key Vault secret names, set the values for sep, "connections_prefix", and "variables_prefix" to "" within AIRFLOW__SECRETS__BACKEND_KWARGS.

Step 4: Deploy configuration

Astro

1. Run the following commands to export your environment variables to Astro.

   astro deployment variable create --deployment-id <your-deployment-id> --load --env .env
   

   In the Astro UI, mark AIRFLOW__SECRETS__BACKEND_KWARGS as Secret. See Set environment variables in the Astro UI.

2. Run the following command to push your updated requirements.txt file to Astro:

   astro deploy --deployment-id <your-deployment-id>
   

3. (Optional) Remove the environment variables from your .env file, or store your .env file so that your credentials are hidden, for example with GitHub secrets.

Remote Execution

1. Run the following command to update your Remote Execution Agent with your new configurations.

helm upgrade astro-agent astronomer/astro-remote-execution-agent -f values.yaml