Create a Redshift connection in Airflow

Learn how to create a Redshift connection in Airflow.

Info

This page has not yet been updated for Airflow 3. The concepts shown are relevant, but some code may need to be updated. If you run any examples, take care to update import statements and watch for any other breaking changes.

Amazon Redshift is a data warehouse product from AWS. Integrating Redshift with Airflow allows you to automate, schedule and monitor a variety of tasks. These tasks include creating, deleting, and resuming a cluster, ingesting or exporting data to and from Redshift, as well as running SQL queries against Redshift.

This document covers two different methods to connect Airflow to Amazon Redshift:

• Using database (DB) user credentials
• Using IAM credentials
• Using IAM role

Tip

If you’re an Astro user, Astronomer recommends using workload identity to authorize to your Deployments to Redshift. This eliminates the need to specify secrets in your Airflow connections or copying credentials file to your Airflow project. See Authorize Deployments to your cloud.

Prerequisites

• The Astro CLI.
• A locally running Astro project.
• Permissions to access to your Redshift cluster. See Using IAM authentication and Authorizing Amazon Redshift to access other AWS services.

Get connection details

Create your connection

Info

Astro users can also create connections using the Astro Environment Manager, which stores connections in an Astro-managed secrets backend. These connections can be shared across multiple deployed and local Airflow environments. See Create Airflow connections in the Astro UI.

How it works

Airflow uses the Amazon Redshift Python Connector to connect to Redshift through the RedshiftSQLHook.

See also

• Apache Airflow Amazon provider package documentation
• Redshift modules in the Airflow Registry
• Import and export Airflow connections using Astro CLI