This guide describes the steps to complete an example Astronomer Software install on Azure to deploy and scale Apache Airflow on a Microsoft Azure Kubernetes Service (AKS) cluster. For full installations, Astronomer recommends platform administrators manually install Astronomer Software using the procedures described in Install Astronomer Software.Documentation Index
Fetch the complete documentation index at: https://astronomer-preview.mintlify.app/llms.txt
Use this file to discover all available pages before exploring further.
Prerequisites
- Git
- Azure CLI
- Kubernetes CLI (kubectl)
- A compatible version of Kubernetes and PostgreSQL as described in Astronomer’s Version compatibility reference
- Helm (minimum v3.6)
- SMTP service and credentials. For example, Mailgun or Sendgrid.
- Permission to create and modify resources on AKS
- Permission to generate a certificate (not self-signed) that covers a defined set of subdomains
- PostgreSQL superuser permissions
- If your organization uses Azure Database for PostgreSQL as the database backend, you need to enable the
pg_trgmextension using the Azure portal or the Azure CLI before you install Astronomer Software. If you don’t enable thepg_trgmextension, the install will fail. For more information about enabling thepg_trgmextension, see PostgreSQL extensions in Azure Database for PostgreSQL - Flexible Server.
Step 1: Choose a base domain
All Astronomer services will be tied to a base domain of your choice, under which you will need the ability to add and edit DNS records. Once created, your Astronomer base domain will be linked to a variety of sub-services that your users will access in a browser to manage, monitor and run Airflow on the platform. For the base domainastro.mydomain.com, for example, here are some corresponding URLs that your users would be able to reach:
- Software UI:
app.astro.mydomain.com - Airflow Deployments:
deployments.astro.mydomain.com/uniquely-generated-airflow-name/airflow - Grafana Dashboard:
grafana.astro.mydomain.com - Kibana Dashboard:
kibana.astro.mydomain.com
Step 2: Configure Azure for Astronomer Deployment
The steps below will walk you through how to:- Create an Azure Resource Group
- Create an AKS Cluster
- Authenticate with your AKS Cluster
Each version of Astronomer Software is compatible with only a particular set of Kubernetes versions. For more information, refer to Astronomer’sVersion compatibility reference.
Create an Azure resource group
A resource group is a collection of related resources for an Azure solution. Your AKS cluster will reside in the resource group you create. Learn more about resource groups here. Login to your Azure account with theaz CLI:
For a list of available locations, run
$ az account list-locations.Create an AKS cluster
Once you’ve created an Azure Resource Group, you can create an AKS cluster using the Azure CLI, Azure PowerShell, or Azure Portal. For instructions, see Microsoft documentation. You can create a cluster with any machine type, but Astronomer recommends using larger nodes and not smaller nodes. Verify your Kubernetes version is supported by Astronomer Software:If you work with multiple Kubernetes environments,
kubectxis an incredibly useful tool for quickly switching between Kubernetes clusters. Learn morehere.Authenticate with your AKS cluster
Run the following command to set your AKS cluster as current context in your kubeconfig. This will configurekubectl to point to your new AKS cluster:
Step 3: Create a Kubernetes namespace
Create a namespace calledastronomer to host the core Astronomer platform:
Step 4: Configure TLS
Astronomer recommends running Astronomer Software on a dedicated domain (BASEDOMAIN) or subdomain (astro.BASEDOMAIN).
In order for users to access the web applications they need to manage Astronomer, you’ll need a TLS certificate that covers the following subdomains:
- Option 1: Obtain a TLS certificate from Let’s Encrypt. Astronomer recommends this option for smaller organizations where your DNS administrator and Kubernetes cluster administrator are either the same person or on the same team.
- Option 2: Request a TLS certificate from your organization’s security team. Astronomer recommends this option for large organizations with their own protocols for generating TLS certificates.
Private CAs support on Azure is only available for clusters running containerd 1.5+, which is available on Kubernetes 1.22+.
Option 1: Create TLS certificates using Let’s Encrypt
Let’s Encrypt is a free and secure certificate authority (CA) service that provides TLS certificates that renew automatically every 90 days. Use this option if you are configuring Astronomer for a smaller organization without a dedicated security team. To set up TLS certificates this way, follow the guidelines in Automatically Renew TLS Certificates Using Let’s Encrypt. Make note of the certificate you create in this setup for Step 5.Option 2: Request a TLS certificate from your security team
If you’re installing Astronomer for a large organization, you’ll need to request a TLS certificate and private key from your enterprise security team. This certificate needs to be valid for theBASEDOMAIN your organization uses for Astronomer, as well as the subdomains listed at the beginning of Step 4. You should be given two .pem files:
- One for your encrypted certificate
- One for your private key
openssl CLI:
X509v3 Subject Alternative Name section of this report includes either a single *.BASEDOMAIN wildcard domain or the subdomains listed at the beginning of Step 4, then the certificate creation was successful.
Depending on your organization, you may receive either a globally trusted certificate or a certificate from a private CA. The certificate from your private CA may include a domain certificate, a root certificate, and/or intermediate certificates, all of which need to be in proper certificate order. To verify certificate order, follow the guidelines below.
Confirm certificate chain order
If your organization is using a private certificate authority, you’ll need to confirm that your certificate chain is ordered correctly. To determine your certificate chain order, run the following command using theopenssl CLI:
- Domain
- Intermediate (optional)
- Root
Step 5: Create a Kubernetes TLS Secret
If you received a globally trusted certificate, such as one generated by Let’s Encrypt, simply run the following command and proceed to Step 6:-
Add the root certificate provided by your security team to an Opaque Kubernetes secret in the Astronomer namespace by running the following command:
The root certificate which you specify here should be the certificate of the authority that signed the Astronomer certificate, rather than the Astronomer certificate itself. This is the same certificate you need to install with all clients to get them to trust your services.The name of the secret file must be
cert.pemfor your certificate to be trusted properly. -
Note the value of
private-root-cafor when you configure your Helm chart in Step 8. You’ll need to additionally specify theprivateCaCertskey-value pair with this value for that step.
Step 6: Configure your SMTP URI
An SMTP service is required for sending and accepting email invites from Astronomer. If you’re running Astronomer Software withpublicSignups disabled (which is the default), you’ll need to configure SMTP as a way for your users to receive and accept invites to the platform through an email invitation. To integrate your SMTP service with Astronomer, fetch your SMTP service’s URI and store it in a Kubernetes secret:
| Provider | Example SMTP URL |
|---|---|
| AWS SES | smtp://AWS_SMTP_Username:AWS_SMTP_Password@email-smtp.us-east-1.amazonaws.com/?requireTLS=true |
| SendGrid | smtps://apikey:SG.sometoken@smtp.sendgrid.net:465/?pool=true |
| Mailgun | smtps://xyz%40example.com:password@smtp.mailgun.org/?pool=true |
| Office365 | smtp://xyz%40example.com:password@smtp.office365.com:587/?requireTLS=true |
| Custom SMTP-relay | smtp://smtp-relay.example.com:25/?ignoreTLS=true |
If there are
/or other escape characters in your username or password, you may need toURL encodethose characters.Step 7: Configure the database
If you’re connecting to an external database, you will need to create a secret namedastronomer-bootstrap to hold your database connection string:
You must URL encode any special characters in your Postgres password.
- If you provision an external database,
postgresqlEnabledshould be set tofalsein Step 8. The in-cluster database should be used only for development or proof-of-concept installations. All production installations should use an external database. - If your organization uses Azure Database for PostgreSQL as the database backend, you need to enable the
pg_trgmextension using the Azure portal or the Azure CLI before you install Astronomer Software. If you don’t enable thepg_trgmextension, the install will fail. For more information about enabling thepg_trgmextension, see PostgreSQL extensions in Azure Database for PostgreSQL - Flexible Server. - If you provision Azure Database for PostgreSQL - Flexible Server, it enforces TLS/SSL and requires that you set
sslmodetopreferin yourvalues.yaml.
Step 8: Configure Your Helm chart
As a next step, create a file named
values.yaml in an empty directory.
For context, this values.yaml file will assume a set of default values for our platform that specify everything from user role definitions to the Airflow images you want to support. As you grow with Astronomer and want to customize the platform to better suit your team and use case, your values.yaml file is the best place to do so.
In the newly created file, copy the example below and replace baseDomain, private-root-ca, /etc/containerd/certs.d, and astronomer.houston.secret with your own values. For more example configuration files, see the Astronomer GitHub.
If there are
/or other escape characters in your username or password, you may need toURL encodethose characters.If you are installing Astronomer in an airgapped environment without access to the public internet, complete all of the setup inInstall in an Airgapped Environmentand then skip directly to Step 10 in this document.
Step 9: Install Astronomer
Now that you have an AKS cluster set up and yourvalues.yaml defined, you’re ready to deploy all components of our platform.
First, run:
--version= flag and use the format 0.36.x. For example, to install Astronomer Software v0.36.0, you specify --version=0.36.0. For more information about the available patch versions, see the Software Release Notes.
When you’re defining <your-platform-release-name>, Astronomer recommends limiting the name to 12 characters to avoid operational issues.
After you run the previous commands, a set of Kubernetes pods are generated in your namespace. These pods power the individual services required to run the Astronomer platform, including the Software UI and Houston API.
Alternative ArgoCD installation
You can install Astronomer with ArgoCD, which is an open source continuous delivery tool for Kubernetes, as an alternative to usinghelm install.
Because ArgoCD doesn’t support sync wave dependencies for app of apps structures, installing Astronomer requires some additional steps compared to the standard ArgoCD workflow:
-
Under the
globalsection of yourvalues.yamlfile, addenableArgoCDAnnotation: true. -
Create a new ArgoCD app. When creating the app, configure the following:
- Path: The filepath of your
values.yamlfile - Namespace: The namespace you want to use for Astronomer
- Cluster: The Kubernetes cluster in which you’re installing Astronomer
- Repository URL:
https://helm.astronomer.io
- Path: The filepath of your
- Sync the ArgoCD app with every component of the Astronomer platform selected. See Sync (Deploy) the Application.
-
Stop the sync when you see that
astronomer-houston-db-migrationshas completed in the Argo UI. -
Sync the application a second time, but this time clear
astronomer-alertmanagerin the Argo UI while keeping all other components selected. Wait for this sync to finish completely. - Sync the ArgoCD app a third time with all Astronomer platform components selected.
Step 10: Verify all pods are up
To verify all pods are up and running, run:Step 11: Configure DNS
Now that you’ve successfully installed Astronomer, a new Load Balancer will have spun up in your Azure account. This Load Balancer routes incoming traffic to our NGINX ingress controller. Runkubectl get svc -n astronomer to view your Load Balancer’s External IP Address, located under the EXTERNAL-IP column for the astronomer-nginx service.
*.astro.mydomain.com, or alternatively create individual A records for the following routes:
Step 12: Verify you can access the Software UI
Go toapp.BASEDOMAIN to see the Software UI.
Consider this your new Airflow control plane. From the Software UI, you’ll be able to both invite and manage users as well as create and monitor Airflow Deployments on the platform.
Step 13: Verify your TLS setup
To check if your TLS certificates were accepted, log in to the Software UI. Then, go toapp.BASEDOMAIN/token and run:
$ astro deploy on a test deployment. Create a deployment in the Software UI, then run:
ImagePullBackoff, check the pod description. If you see an x509 error, ensure that you have:
- Configured containerd’s
config_pathto point to/etc/containerd/certs.d. - Added the
privateCaCertsAddToHostkey-value pairs to your Helm chart.
values.yaml, contact Astronomer support for additional configuration assistance.
What’s next
To help you make the most of Astronomer Software, Astronomer recommends reviewing the following topics:- Renew TLS Certificates on Astronomer Software
- Integrating an Auth System
- Configuring Platform Resources
- Managing Users on Astronomer Software
Astronomer support team
If you have feedback or need help during the installation process, here are some recommended resources:- Community Forum: Search previously asked questions about Airflow + Astronomer FAQs
- Astronomer Support Portal: Platform or Airflow issues