Configure AWS Parameter Store as a secrets backend on Astro Private Cloud

Use AWS Systems Manager Parameter Store as a secrets backend for storing Airflow variables and connections with Astro Private Cloud.

In this section, you’ll learn how to use AWS Systems Manager (SSM) Parameter Store as a secrets backend on Astro Private Cloud.

Prerequisites

• A Deployment.
• The Astro CLI.
• An Astro project initialized with astro dev init.
• Access to AWS SSM Parameter Store.
• A valid AWS Access Key ID and Secret Access Key.

Step 1: Write an Airflow variable or connection to AWS Parameter Store

To start, add an Airflow variable or connection as a secret to Parameter Store for testing. For instructions, see the AWS documentation on how to do so using the AWS Systems Manager Console, the AWS CLI, or Tools for Windows PowerShell.

Variables and connections should live at /airflow/variables and /airflow/connections, respectively. For example, if you’re setting a secret variable with the key my_secret, it should exist at /airflow/connections/my_secret.

Step 2: Set up AWS Parameter Store locally

To test AWS Parameter Store locally, configure it as a secrets backend in your Astro project.

First, install the Airflow provider for Amazon by adding the following to your project’s requirements.txt file:

apache-airflow-providers-amazon

Then, add the following environment variables to your project’s Dockerfile:

# Make sure to replace `<your-aws-key>` and `<your-aws-secret-key>` with your own values.
ENV AWS_ACCESS_KEY_ID="<your-aws-key>"
ENV AWS_SECRET_ACCESS_KEY="<your-aws-secret-key>"
ENV AWS_DEFAULT_REGION="<your-aws-region>"
ENV AIRFLOW__SECRETS__BACKEND=airflow.providers.amazon.aws.secrets.systems_manager.SystemsManagerParameterStoreBackend
ENV AIRFLOW__SECRETS__BACKEND_KWARGS={"connections_prefix": "/airflow/connections", "variables_prefix": "/airflow/variables"}

In the next step, you’ll test that this configuration is valid locally.

If you want to deploy your project to a hosted Git repository before deploying to Astro Private Cloud, be sure to save

<your-aws-key>

and

<your-aws-secret-key>

in a secure manner. When you deploy to Astro Private Cloud, use the UI to set these values as secrets.

If you’d like to reference an AWS profile, you can also add the profile param to ENV AIRFLOW__SECRETS__BACKEND_KWARGS.

To further customize the integration between Airflow and AWS SSM Parameter Store, reference Airflow documentation with the full list of available kwargs.

Step 3: Run an example Dag to test AWS Parameter Store locally

To test Parameter Store, write a simple Dag which calls your secret and add this Dag to your Astro project’s dags directory.

For example, you can use the following Dag to print the value of an Airflow variable to your task logs:

from datetime import datetime

from airflow import DAG
from airflow.models import Variable
from airflow.operators.python import PythonOperator

def print_var():
    my_var = Variable.get("<your-variable-key>")
    print(f'My variable is: {my_var}')

with DAG('example_secrets_dags', start_date=datetime(2022, 1, 1), schedule=None) as dag:

  test_task = PythonOperator(
      task_id='test-task',
      python_callable=print_var,
)

You can do the same for any Airflow connection.

To test your changes:

1. Run astro dev restart to push your changes to your local Airflow environment.

2. In the Airflow UI (http://localhost:8080/admin/), trigger your new Dag.

3. Click on test-task > View Logs. If you ran the example Dag above, you should see the contents of your secret in the task logs:

   {logging_mixin.py:109} INFO - My variable is: my-test-variable
   

Step 4: Deploy to Astro Private Cloud

Once you’ve confirmed that the integration with AWS SSM Parameter Store works locally, you can complete a similar set up with a Deployment on Astro Private Cloud.

1. In the Astro Private Cloud UI, add the same environment variables found in your Dockerfile to your Deployment environment variables. Specify both AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY as secret ensure that your credentials are stored securely.
2. In your Astro project, delete the environment variables from your Dockerfile.
3. Deploy your changes to Astro Private Cloud.

Now, any Airflow variable or connection that you write to AWS SSM Parameter Store can be automatically pulled by any Dag in your Deployment on Astro Private Cloud.