Set up Google Cloud Secret Manager as your secrets backend

Set up Google Cloud Secret Manager as your secrets backend

This topic provides setup steps for configuring Google Cloud Secret Manager as a secrets backend on Astro.

If you use a different secrets backend tool or want to learn the general approach on how to integrate one, see Configure a Secrets Backend.

Prerequisites

• A Deployment.
• The Astro CLI.
• An Astro project.
• Cloud SDK.
• A Google Cloud environment with Secret Manager configured.
• A service account with the Secret Manager Secret Accessor role on Google Cloud.
• (Optional) A JSON service account key for the service account. This is required to provide access to a secrets backend from a local machine, or when you’re not using Workload Identity.
• (Remote Execution Only) Helm installed
• (Remote Execution Only) The values.yaml file from the Register Agents modal in your Deployments>Agents page.

Step 1: Create an Airflow variable or connection in Google Cloud Secret Manager

To start, create an Airflow variable or connection in Google Cloud Secret Manager that you want to store as a secret. You can use the Cloud Console or the gcloud CLI.

Secrets must be formatted such that:

• Airflow variables are set as airflow-variables-<variable-key>.
• Airflow connections are set as airflow-connections-<connection-id>.

For example, to add an Airflow variable with a key my-secret-variable, you run the following gcloud CLI command:

gcloud secrets create airflow-variables-<my-secret-variable> \
    --replication-policy="automatic"

For more information on creating secrets in Google Cloud Secret Manager, read the Google Cloud documentation.

Step 2: Set up GCP Secret Manager locally

Astro

1. Copy the complete JSON service account key for the service account that you want to use to access Secret Manager.

2. Add the following environment variables to your Astro project’s .env file, replacing <your-service-account-key> with the key you copied in Step 1:

   AIRFLOW__SECRETS__BACKEND=airflow.providers.google.cloud.secrets.secret_manager.CloudSecretManagerBackend
   AIRFLOW__SECRETS__BACKEND_KWARGS={"connections_prefix": "airflow-connections", "variables_prefix": "airflow-variables", "gcp_keyfile_dict": "<your-service-account-key>"}
   

3. (Optional) Run Variable.get("<your-variable-key>") to run a dag locally and confirm that your variables are accessible.

Remote Execution

In your Astro project, add the Google Cloud Secret Manager Backend to your project by adding the following to your values.yaml file to set the secrets backend class to use the Vault provider and configure your secrets backend kwargs:

secretBackend: "airflow.providers.google.cloud.secrets.secret_manager.CloudSecretManagerBackend"

commonEnv:
  - name: AIRFLOW__SECRETS__BACKEND_KWARGS
    value: '{"connections_prefix": "airflow-connections", "variables_prefix": "airflow-variables", "gcp_keyfile_dict": "<your-service-account-key>"}'

You need to run the Remote Execution Agent with GCP credentials to fetch from your secrets manager.

Step 3: (Astro Only) Configure Secret Manager on Astro using Workload Identity (Recommended)

1. Set up Workload Identity for your Airflow Deployment. See Connect Astro to GCP data sources.

2. Run the following commands to set the secrets backend for your Astro Deployment:

   $ astro deployment variable create --deployment-id <your-deployment-id> AIRFLOW__SECRETS__BACKEND=airflow.providers.google.cloud.secrets.secret_manager.CloudSecretManagerBackend
   
   $ astro deployment variable create --deployment-id <your-deployment-id> AIRFLOW__SECRETS__BACKEND_KWARGS={"connections_prefix": "airflow-connections", "variables_prefix": "airflow-variables", "project_id": "<your-secret-manager-project-id>"}
   

3. (Optional) Remove the environment variables from your .env file or store your .env file in a safe location to protect your credentials in AIRFLOW__SECRETS__BACKEND_KWARGS.

To ensure the security of secrets, the .env variable is only available in your local environment and not in the Astro UI . See Set Environment Variables Locally.

Step 4: Configure Secret Manager on Astro using a service account JSON key file

Astro

1. Set up the Secret Manager locally. See Set up GCP Secret Manager locally.

2. Run the following command to set the SECRET_VAR_SERVICE_ACCOUNT environment variable on your Astro Deployment:

   astro deployment variable create --deployment-id <your-deployment-id> SECRET_VAR_SERVICE_ACCOUNT="<your-service-account-key>" --secret
   

3. (Optional) Remove the environment variables from your .env file or store your .env file in a safe location to protect your credentials in AIRFLOW__SECRETS__BACKEND_KWARGS.

Remote Execution

1. Run the following command to update your Remote Execution Agent with your new configurations.

helm upgrade astro-agent astronomer/astro-remote-execution-agent -f values.yaml