Configure AWS Secrets Manager as a secrets backend on Astro Private Cloud

Use AWS Secrets Manager as a secrets backend for storing Airflow variables and connections with Astro Private Cloud.

Prerequisites

• A Deployment.
• The Astro CLI.
• An Astro project initialized with astro dev init.
• Access to AWS Secrets Manager.
• A valid AWS Access Key ID and Secret Access Key.

Step 1: Write an Airflow variable or connection to AWS Secrets Manager

To start, add an Airflow variable or connection as a secret to Secrets Manager for testing. For instructions, see the AWS documentation on how to do so using the AWS Secrets Manager Console, CLI or SDK.

Variables and connections should live at /airflow/variables and /airflow/connections, respectively. For example, if you’re setting a secret variable with the key my_secret, it should exist at /airflow/connections/my_secret.

Step 2: Set up AWS Secrets Manager locally

To test AWS Secrets Manager locally, configure it as a secrets backend in your Astro project.

First, install the Airflow provider for Amazon by adding the following to your project’s requirements.txt file:

apache-airflow-providers-amazon

Then, add the following environment variables to your project’s Dockerfile:

# Make sure to replace `<your-aws-key>` and `<your-aws-secret-key>` with your own values.
ENV AWS_ACCESS_KEY_ID="<your-aws-key>"
ENV AWS_SECRET_ACCESS_KEY="<your-aws-secret-key>"
ENV AWS_DEFAULT_REGION="<your-aws-region>"
ENV AIRFLOW__SECRETS__BACKEND=airflow.providers.amazon.aws.secrets.secrets_manager.SecretsManagerBackend
ENV AIRFLOW__SECRETS__BACKEND_KWARGS={"connections_prefix": "airflow/connections", "variables_prefix": "airflow/variables"}

In the next step, you’ll test that this configuration is valid locally.

If you want to deploy your project to a hosted Git repository before deploying to Astro Private Cloud, be sure to save

<your-aws-key>

and

<your-aws-secret-key>

in a secure manner. When you deploy to Astro Private Cloud, use the UI to set these values as secrets.

If you’d like to reference an AWS profile, you can also add the profile param to ENV AIRFLOW__SECRETS__BACKEND_KWARGS.

To further customize the integration between Airflow and AWS Secrets Manager, reference Airflow documentation with the full list of available kwargs.

Step 3: Run an example Dag to test AWS Secrets Manager locally

To test Secrets Manager, write a simple Dag which calls your secret and add this Dag to your Astro project’s dags directory.

For example, you can use the following Dag to print the value of an Airflow variable to your task logs:

from datetime import datetime

from airflow import DAG
from airflow.models import Variable
from airflow.operators.python import PythonOperator

def print_var():
    my_var = Variable.get("<your-variable-key>")
    print(f'My variable is: {my_var}')

with DAG('example_secrets_dags', start_date=datetime(2022, 1, 1), schedule=None) as dag:

  test_task = PythonOperator(
      task_id='test-task',
      python_callable=print_var,
)

To test your changes:

1. Run astro dev restart to push your changes to your local Airflow environment.

2. In the Airflow UI (http://localhost:8080/admin/), trigger your new dag.

3. Click on test-task > View Logs. If you ran the example Dag above, you should see the contents of your secret in the task logs:

   {logging_mixin.py:109} INFO - My variable is: my-test-variable
   

Step 4: Deploy to Astro Private Cloud

Once you’ve confirmed that the integration with AWS Secrets Manager works locally, you can complete a similar set-up with a Deployment on Astro Private Cloud.

1. In the Astro Private Cloud UI, add the same environment variables found in your Dockerfile to your Deployment environment variables. Specify both AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY as secret ensure that your credentials are stored securely.
2. In your Astro project, delete the environment variables from your Dockerfile.
3. Deploy your changes to Astro Private Cloud.

Now, any Airflow variable or connection that you write to AWS Secrets Manager can be automatically pulled by any Dag in your Deployment on Astro Private Cloud.