# Astronomer Docs ## Docs - [Create an Agent API token](https://astronomer-preview.mintlify.app/api-reference/agenttoken/create-an-agent-api-token.md): Create an Agent API token for remote workers. - [Delete an Agent API Token](https://astronomer-preview.mintlify.app/api-reference/agenttoken/delete-an-agent-api-token.md): Delete an Agent API Token. - [Get an Agent API Token](https://astronomer-preview.mintlify.app/api-reference/agenttoken/get-an-agent-api-token.md): Get an Agent API Token. - [List Agent API Tokens](https://astronomer-preview.mintlify.app/api-reference/agenttoken/list-agent-api-tokens.md): List Agent API Tokens. - [Create an allowed IP address range](https://astronomer-preview.mintlify.app/api-reference/allowedipaddressrange/create-an-allowed-ip-address-range.md): Create an allowed IP address range that constrains which IP addresses can be used to interact with your Astro Organization using APIs. - [Delete an allowed IP address range](https://astronomer-preview.mintlify.app/api-reference/allowedipaddressrange/delete-an-allowed-ip-address-range.md): Delete an allowed IP address range. - [List allowed IP address ranges](https://astronomer-preview.mintlify.app/api-reference/allowedipaddressrange/list-allowed-ip-address-ranges.md): List allowed IP address ranges. - [Create an API token](https://astronomer-preview.mintlify.app/api-reference/apitoken/create-an-api-token.md): Create an API token. An API token is an alphanumeric token that grants programmatic access to Astro for automated workflows. An API token can be scoped to an Organization or a Workspace. - [Delete an API token](https://astronomer-preview.mintlify.app/api-reference/apitoken/delete-an-api-token.md): Delete an API token. When you delete an API token, make sure that no existing automation workflows are using it. After it's deleted, an API token cannot be recovered. - [Get an API token](https://astronomer-preview.mintlify.app/api-reference/apitoken/get-an-api-token.md): Retrieve information about a specific API token. - [List API tokens](https://astronomer-preview.mintlify.app/api-reference/apitoken/list-api-tokens.md): List information about all API tokens from an Organization. Filters on Workspace when Workspace ID is provided. When `includeOnlyOrganizationTokens` is `true`, only Organization API tokens are returned. - [Rotate API token](https://astronomer-preview.mintlify.app/api-reference/apitoken/rotate-api-token.md): Rotate an API token. Creates a new API token and invalidates the one you specify. Any workflows using the previous value stop working. - [Update an API token](https://astronomer-preview.mintlify.app/api-reference/apitoken/update-an-api-token.md): Update the name and description of an API token. - [Update API token roles](https://astronomer-preview.mintlify.app/api-reference/apitoken/update-api-token-roles.md): Update Workspace and Organization roles for an API token. - [List authorization permission groups](https://astronomer-preview.mintlify.app/api-reference/authorization/list-authorization-permission-groups.md): List the available permissions you can grant to a custom role. - [Create a cluster](https://astronomer-preview.mintlify.app/api-reference/cluster/create-a-cluster.md): Create a cluster in the Organization. An Astro cluster is a Kubernetes cluster that hosts the infrastructure required to run Deployments. - [Delete a cluster](https://astronomer-preview.mintlify.app/api-reference/cluster/delete-a-cluster.md): Delete a cluster. - [Get a cluster](https://astronomer-preview.mintlify.app/api-reference/cluster/get-a-cluster.md): Retrieve details about a cluster. - [List clusters](https://astronomer-preview.mintlify.app/api-reference/cluster/list-clusters.md): List clusters in an Organization. - [Update a cluster](https://astronomer-preview.mintlify.app/api-reference/cluster/update-a-cluster.md): Update a cluster in the Organization. - [Create a Deploy](https://astronomer-preview.mintlify.app/api-reference/deploy/create-a-deploy.md): Create a new Deploy. A Deploy represents an intent to deploy new DAG code to an Astro Deployment. - [Deploy rollback of a deployment](https://astronomer-preview.mintlify.app/api-reference/deploy/deploy-rollback-of-a-deployment.md): Deploy rollback of a deployment - [Finalize a deploy](https://astronomer-preview.mintlify.app/api-reference/deploy/finalize-a-deploy.md): Finalize a deploy that you initialized using the `deploys` endpoint. - [Get a deploy for a deployment](https://astronomer-preview.mintlify.app/api-reference/deploy/get-a-deploy-for-a-deployment.md): Get a deploy for a deployment - [List deploys for a deployment](https://astronomer-preview.mintlify.app/api-reference/deploy/list-deploys-for-a-deployment.md): List deploys for a deployment - [Update a Deploy](https://astronomer-preview.mintlify.app/api-reference/deploy/update-a-deploy.md): Update an existing Deploy. A Deploy represents an intent to deploy new DAG code to an Astro Deployment. - [Configure a hibernation override for a deployment](https://astronomer-preview.mintlify.app/api-reference/deployment/configure-a-hibernation-override-for-a-deployment.md) - [Create a Deployment](https://astronomer-preview.mintlify.app/api-reference/deployment/create-a-deployment.md): Create a Deployment in the Organization. An Astro Deployment is an Airflow environment that is powered by all core Airflow components. - [Delete a Deployment](https://astronomer-preview.mintlify.app/api-reference/deployment/delete-a-deployment.md): Delete a Deployment from an Organization. - [Delete a hibernation override for a deployment](https://astronomer-preview.mintlify.app/api-reference/deployment/delete-a-hibernation-override-for-a-deployment.md) - [Get a Deployment](https://astronomer-preview.mintlify.app/api-reference/deployment/get-a-deployment.md): Retrieve details about a Deployment. - [Get Deployment logs](https://astronomer-preview.mintlify.app/api-reference/deployment/get-deployment-logs.md): Get logs for an Astro Deployment. You must specify at least one log source. - [List Deployments](https://astronomer-preview.mintlify.app/api-reference/deployment/list-deployments.md): List Deployments in an Organization. - [Update a Deployment](https://astronomer-preview.mintlify.app/api-reference/deployment/update-a-deployment.md): Update a Deployment in the Organization. - [Create a new environment object](https://astronomer-preview.mintlify.app/api-reference/environment/create-a-new-environment-object.md): Create an environment object for a Deployment or Workspace. An environment object represents a connection, Airflow variable, or metrics export resource that is managed by the Astro Environment Manager. - [Delete an environment object](https://astronomer-preview.mintlify.app/api-reference/environment/delete-an-environment-object.md): Delete an environment object from a Deployment or Workspace. - [Exclude linking an environment object](https://astronomer-preview.mintlify.app/api-reference/environment/exclude-linking-an-environment-object.md): Exclude a specific Deployment from linking to an environment object created at the Workspace level. - [Get environment object](https://astronomer-preview.mintlify.app/api-reference/environment/get-environment-object.md): Retrieve details about an environment object. - [List environment objects](https://astronomer-preview.mintlify.app/api-reference/environment/list-environment-objects.md): List environment objects in a Workspace or Deployment. - [Update an environment object](https://astronomer-preview.mintlify.app/api-reference/environment/update-an-environment-object.md): Update an existing Deployment level or Workspace level environment object. - [Create a user invitation](https://astronomer-preview.mintlify.app/api-reference/invite/create-a-user-invitation.md): Invite a user to an Organization. - [Delete a user invite](https://astronomer-preview.mintlify.app/api-reference/invite/delete-a-user-invite.md): Delete an existing user invite. - [Get cluster options](https://astronomer-preview.mintlify.app/api-reference/options/get-cluster-options.md): Get all possible options for configuring a cluster. - [Get Deployment options](https://astronomer-preview.mintlify.app/api-reference/options/get-deployment-options.md): Get the options available for configuring a Deployment. - [Get an Organization](https://astronomer-preview.mintlify.app/api-reference/organization/get-an-organization.md): Retrieve information about a specific Organization. - [Get Organization audit logs](https://astronomer-preview.mintlify.app/api-reference/organization/get-organization-audit-logs.md): Retrieve logs across a specified time period for an Organization. - [List Organizations](https://astronomer-preview.mintlify.app/api-reference/organization/list-organizations.md): List the details about all Organizations that you have access to. Requires using a personal access token (PAT) for authentication. - [Update an Organization](https://astronomer-preview.mintlify.app/api-reference/organization/update-an-organization.md): Update an Organization's details. - [Create a custom role](https://astronomer-preview.mintlify.app/api-reference/role/create-a-custom-role.md): Create a custom role that you can assign to users, Teams, and API tokens. - [Delete a custom role](https://astronomer-preview.mintlify.app/api-reference/role/delete-a-custom-role.md): Delete a custom role. - [Get a custom role](https://astronomer-preview.mintlify.app/api-reference/role/get-a-custom-role.md): Get details about a custom role. - [Get role templates](https://astronomer-preview.mintlify.app/api-reference/role/get-role-templates.md): Get a list of available role templates in an Organization. A role template can be used as the basis for creating a new custom role. - [List roles](https://astronomer-preview.mintlify.app/api-reference/role/list-roles.md): List available user roles in an Organization. - [Update custom role](https://astronomer-preview.mintlify.app/api-reference/role/update-custom-role.md): Update the metadata or included permissions for a custom role. - [Add members to a team](https://astronomer-preview.mintlify.app/api-reference/team/add-members-to-a-team.md): Add members to a team - [Create a Team](https://astronomer-preview.mintlify.app/api-reference/team/create-a-team.md): Create a Team in an Organization. A Team is a group of users that have the same set of permissions in an Organization or Workspace. - [Delete a Team](https://astronomer-preview.mintlify.app/api-reference/team/delete-a-team.md): Delete a Team. Deleting a Team will remove all permissions associated with the Team. Users that previously belonged to the Team will no longer have these permissions. - [Get a Team](https://astronomer-preview.mintlify.app/api-reference/team/get-a-team.md): Retrieve details about a specific Team. - [List Team members](https://astronomer-preview.mintlify.app/api-reference/team/list-team-members.md): List the details about all users that belong to a specific Team. - [List Teams](https://astronomer-preview.mintlify.app/api-reference/team/list-teams.md): List all Teams in an Organization. - [Remove Team member](https://astronomer-preview.mintlify.app/api-reference/team/remove-team-member.md): Remove a user from a Team. The user loses all permissions associated with the Team. - [Update a Team](https://astronomer-preview.mintlify.app/api-reference/team/update-a-team.md): Update a Team's details. - [Update Team roles](https://astronomer-preview.mintlify.app/api-reference/team/update-team-roles.md): Update Organization and Workspace roles for a Team. - [Get current user](https://astronomer-preview.mintlify.app/api-reference/user/get-current-user.md): Get the authenticated user's profile and roles. - [Get user information](https://astronomer-preview.mintlify.app/api-reference/user/get-user-information.md): Retrieve user information about a specific user account. - [List teams for a user](https://astronomer-preview.mintlify.app/api-reference/user/list-teams-for-a-user.md): List the teams a user belongs to within an Organization. - [List users in an Organization](https://astronomer-preview.mintlify.app/api-reference/user/list-users-in-an-organization.md): List users in an Organization or a specific Workspace within an Organization. - [Update a user's roles](https://astronomer-preview.mintlify.app/api-reference/user/update-a-users-roles.md): Update Organization and Workspace roles for a user. - [Create Workspace](https://astronomer-preview.mintlify.app/api-reference/workspace/create-workspace.md): Create a Workspace. - [Delete Workspace](https://astronomer-preview.mintlify.app/api-reference/workspace/delete-workspace.md): Delete a Workspace. - [Get Workspace](https://astronomer-preview.mintlify.app/api-reference/workspace/get-workspace.md): Get information about a Workspace. - [List Workspaces](https://astronomer-preview.mintlify.app/api-reference/workspace/list-workspaces.md): List Workspaces in an Organization - [Update Workspace](https://astronomer-preview.mintlify.app/api-reference/workspace/update-workspace.md): Update a Workspace. - [Remote Execution Agent image reference](https://astronomer-preview.mintlify.app/docs/astro/agent-images.md): Reference of images included in each Astro Remote Execution Agent release. - [Remote Execution Agent maintenance policy](https://astronomer-preview.mintlify.app/docs/astro/agent-maintenance-policy.md): Remote Execution Agent maintenance policy - [Remote Execution Agent release notes](https://astronomer-preview.mintlify.app/docs/astro/agent-release-notes.md): A summary of the latest Remote Execution Agent features and functionality. - [Using the Airflow REST API with Astro](https://astronomer-preview.mintlify.app/docs/astro/airflow-api.md): Use the Airflow REST API in Astro to trigger Dag runs, list Dags, manage task instances, and automate workflows programmatically. Includes authentication setup and code examples. - [Configure Airflow email notifications on Astro](https://astronomer-preview.mintlify.app/docs/astro/airflow-email-notifications.md): Set up email notifications for Airflow task successes and failures. - [Airflow 3 features](https://astronomer-preview.mintlify.app/docs/astro/airflow3/features-af3.md): Features of Airflow 3 - [Upgrade an Astro project to Airflow 3](https://astronomer-preview.mintlify.app/docs/astro/airflow3/upgrade-af3.md): Learn how to upgrade your Astro project from Apache Airflow 2 to Apache Airflow 3. - [Set up Astro alerts](https://astronomer-preview.mintlify.app/docs/astro/alerts.md): Set up Astro alerts - [Add Astro domains to your network](https://astronomer-preview.mintlify.app/docs/astro/allowlist-domains.md): A list of Astro domains to add to your organization's network allowlist. - [Deploy to Astro with the API](https://astronomer-preview.mintlify.app/docs/astro/api/deploy-with-api.md): Deploy to Astro with the API - [Get started with the Astro API](https://astronomer-preview.mintlify.app/docs/astro/api/get-started.md): Run a few simple commands to learn how to use the Astro API. - [Migrate to v1](https://astronomer-preview.mintlify.app/docs/astro/api/migrate-v1-api.md): Migrate to v1 - [Astro API overview](https://astronomer-preview.mintlify.app/docs/astro/api/overview.md): Astro API overview - [Astro API versioning, maintenance, and support](https://astronomer-preview.mintlify.app/docs/astro/api/versioning-and-support.md): Learn how Astronomer versions and maintains the Astro API. - [Assets overview](https://astronomer-preview.mintlify.app/docs/astro/assets-overview.md): About data assets in Astro Observe. - [Event timeline](https://astronomer-preview.mintlify.app/docs/astro/assets-timeline.md): View asset event timeline in Astro Observe. - [About Astro](https://astronomer-preview.mintlify.app/docs/astro/astro-architecture.md): Learn about how Astro is structured to maximize the power of Apache Airflow. - [Astro executor](https://astronomer-preview.mintlify.app/docs/astro/astro-executor.md): Astro executor - [Astro glossary](https://astronomer-preview.mintlify.app/docs/astro/astro-glossary.md): A quick reference for terms you'll encounter on Astro. - [Astro MCP servers](https://astronomer-preview.mintlify.app/docs/astro/astro-mcp-server.md): Connect AI tools to Astro and Airflow using Model Context Protocol (MCP) servers - [Astro Observe overview](https://astronomer-preview.mintlify.app/docs/astro/astro-observe.md): Gain insight into the health and performance of your data pipelines and lifecycle. - [Book Astro office hours](https://astronomer-preview.mintlify.app/docs/astro/astro-office-hours.md): Attend office hours with Astronomer data engineers to ask non-support questions about Airflow. - [Submit a support request](https://astronomer-preview.mintlify.app/docs/astro/astro-support.md): Get Astro support when you need it. - [Export Astro audit logs](https://astronomer-preview.mintlify.app/docs/astro/audit-logs.md): Export Astro audit logs - [Authorize an Astro Deployment to cloud resources using workload identity](https://astronomer-preview.mintlify.app/docs/astro/authorize-deployments-to-your-cloud.md): Give Astro Deployments access to your cloud resources using a Kubernetes workload identity - [Authorize Workspaces to a cluster](https://astronomer-preview.mintlify.app/docs/astro/authorize-workspaces-to-a-cluster.md): Learn how to configure a cluster so that only specific Workspaces can use it. - [Authenticate an automation tool to Astro](https://astronomer-preview.mintlify.app/docs/astro/automation-authentication.md): Learn about all possible ways that you can authenticate your automation tool to Astro. - [Automate actions on Astro](https://astronomer-preview.mintlify.app/docs/astro/automation-overview.md): Learn how you can automate various actions on Astro to quickly build and manage your data ecosystem. - [Airflow Edge Cases on Astro](https://astronomer-preview.mintlify.app/docs/astro/best-practices/airflow-edge-cases.md): Edge cases when running Airflow that give unintuitive behavior and how to avoid them - [When to use Airflow or Astro alerts for your pipelines on Astro](https://astronomer-preview.mintlify.app/docs/astro/best-practices/airflow-vs-astro-alerts.md): When to use Airflow or Astro alerts for your pipelines on Astro - [Astro API best practices](https://astronomer-preview.mintlify.app/docs/astro/best-practices/astro-api-best-practices.md): Best practices for developing with the Astro API - [Manage Astro connections in branch-based deploy workflows](https://astronomer-preview.mintlify.app/docs/astro/best-practices/connections-branch-deploys.md): Manage Astro connections in branch-based deploy workflows - [Cross-deployment dependencies](https://astronomer-preview.mintlify.app/docs/astro/best-practices/cross-deployment-dependencies.md): How to implement dependencies between your Airflow deployments. - [Dag writing on Astro](https://astronomer-preview.mintlify.app/docs/astro/best-practices/dag-writing-on-astro.md): Dag writing on Astro - [Use Git submodules with an Astro project](https://astronomer-preview.mintlify.app/docs/astro/best-practices/git-submodules.md): Combine code from multiple repositories into a single Astro project using Git submodules. - [Manage development Deployments on Astro](https://astronomer-preview.mintlify.app/docs/astro/best-practices/manage-dev-deployments.md): Manage development Deployments on Astro - [Manage resources on Astro](https://astronomer-preview.mintlify.app/docs/astro/best-practices/manage-resources.md): Manage resources on Astro - [Choose a code repository strategy](https://astronomer-preview.mintlify.app/docs/astro/best-practices/repo-structure.md): Choose a code repository strategy - [Best practices for rightsizing Airflow resources on Astro](https://astronomer-preview.mintlify.app/docs/astro/best-practices/rightsize-airflow-on-astro.md): Use Astro Deployment metrics to rightsize your Airflow resources on Astro. - [Best practices for upgrading Astro Runtime on Astro](https://astronomer-preview.mintlify.app/docs/astro/best-practices/upgrading-astro-runtime.md): A list of best practices to ensure that your Astro Runtime upgrades are safe and easy. - [Configure the Celery executor](https://astronomer-preview.mintlify.app/docs/astro/celery-executor.md): Configure the Celery executor - [Astro CI/CD templates for AWS CodeBuild](https://astronomer-preview.mintlify.app/docs/astro/ci-cd-templates/aws-codebuild.md): Use pre-built Astronomer CI/CD templates to automate deploying Apache Airflow dags to Astro using AWS CodeBuild. - [Deploy dags from an AWS S3 bucket to Astro using AWS Lambda](https://astronomer-preview.mintlify.app/docs/astro/ci-cd-templates/aws-s3.md): Use pre-built Astronomer CI/CD templates to automate deploying Apache Airflow dags to Astro using AWS S3 and Lambda. - [Astro CI/CD templates for Azure DevOps](https://astronomer-preview.mintlify.app/docs/astro/ci-cd-templates/azure-devops.md): Use pre-built Astronomer CI/CD templates to automate deploying Apache Airflow dags to Astro using Azure DevOps. - [Astro CI/CD templates for Bitbucket](https://astronomer-preview.mintlify.app/docs/astro/ci-cd-templates/bitbucket.md): Use pre-built Astronomer CI/CD templates to automate deploying Apache Airflow dags to Astro using BitBucket. - [Astro CI/CD templates for CircleCI](https://astronomer-preview.mintlify.app/docs/astro/ci-cd-templates/circleci.md): Use pre-built Astronomer CI/CD templates to automate deploying Apache Airflow dags to Astro using CircleCI. - [dbt deploy action for deploying dbt code to Astro](https://astronomer-preview.mintlify.app/docs/astro/ci-cd-templates/dbt-deploy-action.md): Use pre-built Astronomer CI/CD templates to automate deploying dbt code to Astro using GitHub Actions. - [Default deploy action for deploying code to Astro](https://astronomer-preview.mintlify.app/docs/astro/ci-cd-templates/default-deploy-action.md): Use pre-built Astronomer CI/CD templates to automate deploying code to Astro using GitHub Actions. - [Astro CI/CD templates for Drone](https://astronomer-preview.mintlify.app/docs/astro/ci-cd-templates/drone.md): Use pre-built Astronomer CI/CD templates to automate deploying Apache Airflow dags to Astro using Drone CI. - [Deploy Dags from Google Cloud Storage to Astro](https://astronomer-preview.mintlify.app/docs/astro/ci-cd-templates/gcs.md): Use pre-built Astronomer CI/CD templates to automate deploying Apache Airflow Dags to Astro using Google Cloud Storage. - [GitHub Actions templates for deploying code to preview Deployments on Astro](https://astronomer-preview.mintlify.app/docs/astro/ci-cd-templates/github-actions-deployment-preview.md): Use pre-built Astronomer CI/CD templates to automate deploying Apache Airflow dags to a preview Deployment using GitHub Actions. - [GitHub Actions templates for deploying to Astro from private networks](https://astronomer-preview.mintlify.app/docs/astro/ci-cd-templates/github-actions-private-network.md): Use pre-built Astronomer CI/CD templates to automate deploying Apache Airflow dags to Astro on private networks using GitHub Actions. - [GitHub Actions templates for deploying code to Astro](https://astronomer-preview.mintlify.app/docs/astro/ci-cd-templates/github-actions-template.md): Use pre-built Astronomer CI/CD templates to automate deploying Apache Airflow dags to Astro using GitHub Actions. - [Astro CI/CD templates for GitLab](https://astronomer-preview.mintlify.app/docs/astro/ci-cd-templates/gitlab.md): Use pre-built Astronomer CI/CD templates to automate deploying Apache Airflow dags to Astro using GitLab. - [Astro CI/CD templates for Harness](https://astronomer-preview.mintlify.app/docs/astro/ci-cd-templates/harness.md): Use pre-built Astronomer CI/CD templates to automate deploying Apache Airflow dags to Astro using Harness. - [Astro CI/CD templates for Jenkins](https://astronomer-preview.mintlify.app/docs/astro/ci-cd-templates/jenkins.md): Use pre-built Astronomer CI/CD templates to automate deploying Apache Airflow dags to Astro using Jenkins. - [Template options](https://astronomer-preview.mintlify.app/docs/astro/ci-cd-templates/template-overview.md): Use pre-built templates to get started with automating code deploys - [Set up authentication and single sign-on for Astro](https://astronomer-preview.mintlify.app/docs/astro/configure-idp.md): Configure federated authentication from a variety of third party identity providers on Astro. - [Configure worker queues](https://astronomer-preview.mintlify.app/docs/astro/configure-worker-queues.md): Learn how to create and configure worker queues to create best-fit execution environments for your tasks. - [Create a network connection between Astro and AWS](https://astronomer-preview.mintlify.app/docs/astro/connect-aws.md): Create a network connection to AWS. - [AWS Networking: VPN](https://astronomer-preview.mintlify.app/docs/astro/connect-aws-VPN.md): Create a VPN connection to AWS. - [AWS Networking: Hostname resolution options](https://astronomer-preview.mintlify.app/docs/astro/connect-aws-hostname-resolution.md): Hostname resolution options for AWS. - [AWS Networking: AWS PrivateLink](https://astronomer-preview.mintlify.app/docs/astro/connect-aws-privatelink.md): Create a PrivateLink connection to AWS. - [Connect to a public AWS Endpoint](https://astronomer-preview.mintlify.app/docs/astro/connect-aws-public.md): Create a network connection to a public AWS endpoint. - [AWS Networking: Transit Gateways](https://astronomer-preview.mintlify.app/docs/astro/connect-aws-transit-gateways.md): Create a Transit Gateway connection to AWS. - [AWS Networking: VPC Peering](https://astronomer-preview.mintlify.app/docs/astro/connect-aws-vpc-peering.md): Create a VPC peering network connection to AWS. - [Create a network connection between Astro and Azure](https://astronomer-preview.mintlify.app/docs/astro/connect-azure.md): Create a network connection to Microsoft Azure. - [Azure Networking: Hostname resolution options](https://astronomer-preview.mintlify.app/docs/astro/connect-azure-hostname-resolution.md): Hostname resolution options for Azure. - [Azure Networking: Azure PrivateLink](https://astronomer-preview.mintlify.app/docs/astro/connect-azure-private-link.md): Create a PrivateLink connection to Azure. - [Create a connection between Astro and Azure public endpoint](https://astronomer-preview.mintlify.app/docs/astro/connect-azure-public.md): Create a network connection to a public endpoint on Microsoft Azure. - [Azure Networking: VHub Peering](https://astronomer-preview.mintlify.app/docs/astro/connect-azure-vhub-peering.md): Create a VHub peering network connection to Azure. - [Azure Networking: VNet Peering](https://astronomer-preview.mintlify.app/docs/astro/connect-azure-vnet-peering.md): Create a VNet peering network connection to Azure. - [Azure Networking: VPN](https://astronomer-preview.mintlify.app/docs/astro/connect-azure-vpn.md): Create a VPN connection to Azure. - [Create a network connection between Astro and GCP](https://astronomer-preview.mintlify.app/docs/astro/connect-gcp.md): Create a network connection to Google Cloud Platform. - [Create Airflow connections in the Astro UI](https://astronomer-preview.mintlify.app/docs/astro/create-and-link-connections.md): Create Airflow connections and link them to multiple Deployments in the Astro Environment Manager. - [Create environment variables in the Astro UI](https://astronomer-preview.mintlify.app/docs/astro/create-and-link-environment-variables.md): Create environment variables and link them to multiple Deployments in the Astro Environment Manager. - [Create Airflow variables in the Astro UI](https://astronomer-preview.mintlify.app/docs/astro/create-and-link-variables.md): Create Airflow variables and link them to multiple Deployments in the Astro Environment Manager. - [Create and use data products with Astro Observe](https://astronomer-preview.mintlify.app/docs/astro/create-data-products.md): Create observability tools on Astro to gain insight into the health and performance of your data pipelines and lifecycle. - [Create a dedicated Astro cluster](https://astronomer-preview.mintlify.app/docs/astro/create-dedicated-cluster.md): Create a dedicated Astro cluster - [Create a Deployment](https://astronomer-preview.mintlify.app/docs/astro/create-deployment.md): Learn how to create an Astro Deployment. After you’ve created a Deployment, you can deploy dags to it from the Astro command-line interface (CLI), or from a continuous integration and continuous delivery (CI/CD) pipeline. - [Configure Customer Managed Egress](https://astronomer-preview.mintlify.app/docs/astro/customer-managed-egress.md): Configure Customer Managed Egress - [Create and assign custom Deployment roles](https://astronomer-preview.mintlify.app/docs/astro/customize-deployment-roles.md): Customize your users' permissions for Airflow environments on Astro. - [Dag-level access control](https://astronomer-preview.mintlify.app/docs/astro/dag-level-access-control.md): Assign fine-grained, per-Dag roles to users, Teams, and API tokens on Astro. - [Dag versioning](https://astronomer-preview.mintlify.app/docs/astro/dag-versioning.md): Dag versioning - [Write and run dags on Astro](https://astronomer-preview.mintlify.app/docs/astro/dags-overview.md): Learn how to use code-level features specific to Astro in your Apache Airflow dags. - [Data protection](https://astronomer-preview.mintlify.app/docs/astro/data-protection.md): Learn how Astronomer uses encryption to protect clusters and data. - [Deploy code to Astro](https://astronomer-preview.mintlify.app/docs/astro/deploy-code.md): Learn about the different ways you can deploy code to Astro. - [Deploy Dags to Astro](https://astronomer-preview.mintlify.app/docs/astro/deploy-dags.md): Learn about the different ways you can deploy code to Astro. - [Deploy dbt projects to Astro](https://astronomer-preview.mintlify.app/docs/astro/deploy-dbt-project.md): Learn how to deploy and run dbt projects with Apache Airflow on Astro. - [Deploy code with the Astro GitHub integration](https://astronomer-preview.mintlify.app/docs/astro/deploy-github-integration.md): Learn how to automatically deploy Apache Airflow code to Astro from GitHub with a built-in integration. - [Roll back to previous deploys using deploy histories](https://astronomer-preview.mintlify.app/docs/astro/deploy-history.md): View a historical record of code deploys to an Astro Deployment and roll back to specific deploys when something goes wrong. - [Deploy an Astro project as an image](https://astronomer-preview.mintlify.app/docs/astro/deploy-project-image.md): Deploy a complete Astro project to a Deployment as a Docker image. - [Initialize and deploy Remote Execution projects](https://astronomer-preview.mintlify.app/docs/astro/deploy-project-remote-execution.md): Initialize and deploy an Astro project for Remote Execution. - [Create and manage Deployment API tokens](https://astronomer-preview.mintlify.app/docs/astro/deployment-api-tokens.md): Use Deployment API tokens to automate code deploys and configuration changes to a Deployment. - [Deployment details](https://astronomer-preview.mintlify.app/docs/astro/deployment-details.md): Edit information about your Deployment, like metadata settings, observability settings, and user access settings. - [Deployment file reference](https://astronomer-preview.mintlify.app/docs/astro/deployment-file-reference.md): View all possible values that you can include in a Deployment file when managing Deployments as code. - [View and address Deployment health incidents](https://astronomer-preview.mintlify.app/docs/astro/deployment-health-incidents.md): A list of all possible Deployment incident types and steps for resolving each one. - [View metrics for Astro Deployments](https://astronomer-preview.mintlify.app/docs/astro/deployment-metrics.md): Learn how to monitor Deployment performance, health, and total task volume in the Astro UI. These metrics can help you with resource allocation and issue troubleshooting. - [Configure Deployment resources](https://astronomer-preview.mintlify.app/docs/astro/deployment-resources.md): Configure your Deployment resource settings to optimize Deployment performance. - [Custom role permissions reference](https://astronomer-preview.mintlify.app/docs/astro/deployment-role-reference.md): Learn about each possible permission that you can assign to custom Deployment and Dag roles. - [Overview](https://astronomer-preview.mintlify.app/docs/astro/deployment-settings.md): View details and options for your Deployment. - [Astro deprecations](https://astronomer-preview.mintlify.app/docs/astro/deprecations.md): Learn about Astro feature deprecations - [Disaster recovery](https://astronomer-preview.mintlify.app/docs/astro/disaster-recovery.md): Learn how Astronomer handles disaster recovery scenarios and how to set up self-service cross-region disaster recovery for dedicated AWS clusters. - [Trigger failover and failback](https://astronomer-preview.mintlify.app/docs/astro/disaster-recovery-failover.md): Trigger a failover to your secondary cluster or fail back to the primary cluster after it recovers. - [Prepare for disaster recovery](https://astronomer-preview.mintlify.app/docs/astro/disaster-recovery-prepare.md): Prepare your Astro environment for cross-region disaster recovery, including networking, workload identity, and Dag configuration. - [Set up disaster recovery](https://astronomer-preview.mintlify.app/docs/astro/disaster-recovery-setup.md): Enable cross-region disaster recovery on new or existing dedicated AWS clusters. - [Environment variables](https://astronomer-preview.mintlify.app/docs/astro/environment-variables.md): Overview of environment variables on Astro - [Execution mode](https://astronomer-preview.mintlify.app/docs/astro/execution-mode.md): Overview of Remote and Hosted execution - [Manage Airflow executors on Astro](https://astronomer-preview.mintlify.app/docs/astro/executors-overview.md): Manage Airflow executors on Astro - [Export logs to AWS CloudWatch](https://astronomer-preview.mintlify.app/docs/astro/export-cloudwatch.md): Configure your Deployment to forward observability data to your AWS CloudWatch instance. - [Export metrics and logs to Datadog](https://astronomer-preview.mintlify.app/docs/astro/export-datadog.md): Configure your Deployment to forward observability data to your Datadog instance. - [Export metrics from Astro](https://astronomer-preview.mintlify.app/docs/astro/export-metrics.md): Export metrics from Astro - [Export logs to a Secondary GCS Bucket](https://astronomer-preview.mintlify.app/docs/astro/export-secondary-gcs-bucket.md): Configure your Deployment to forward Airflow task logs to a secondary GCS bucket in GCP. - [Export logs to a Secondary S3 Bucket](https://astronomer-preview.mintlify.app/docs/astro/export-secondary-s3-bucket.md): Configure your Deployment to forward Airflow task logs to a secondary AWS S3 bucket. - [Export logs to a Secondary WASB Container](https://astronomer-preview.mintlify.app/docs/astro/export-secondary-wasb.md): Configure your Deployment to forward Airflow task logs to a secondary Azure Storage (WASB) container. - [Astro feature previews](https://astronomer-preview.mintlify.app/docs/astro/feature-previews.md): Learn about Astronomer feature previews. - [Astro features](https://astronomer-preview.mintlify.app/docs/astro/features.md): Set environment variables on Astro to specify Airflow configurations and custom logic. - [Run your first dag with the Astro CLI](https://astronomer-preview.mintlify.app/docs/astro/first-dag-cli.md): Learn how to run your first Apache Airflow® dag on Astro with the Astro CLI. - [Run your first dag with GitHub Actions](https://astronomer-preview.mintlify.app/docs/astro/first-dag-github-actions.md): Learn how to run your first Apache Airflow® dag on Astro with GitHub Actions. - [Run your first dag on Astro](https://astronomer-preview.mintlify.app/docs/astro/first-dag-onboarding.md): Learn how to run your first Apache Airflow® dag on Astro when you sign up for Astro. - [GDPR compliance](https://astronomer-preview.mintlify.app/docs/astro/gdpr-compliance.md): Learn how Astronomer and Astro are GDPR compliant. - [HIPAA compliance](https://astronomer-preview.mintlify.app/docs/astro/hipaa-compliance.md): Learn how to achieve HIPAA compliance on Astro. - [Author dags with the Astro IDE](https://astronomer-preview.mintlify.app/docs/astro/ide-author-dags.md): Author dags with the Astro IDE - [Blueprint](https://astronomer-preview.mintlify.app/docs/astro/ide-blueprint.md): Use Blueprint to create Airflow Dags from platform-defined templates in the Astro IDE, without writing Airflow code. - [Deploy code from the Astro IDE](https://astronomer-preview.mintlify.app/docs/astro/ide-deploy.md): Deploy code from the Astro IDE - [Deploy code from the Astro IDE to Git](https://astronomer-preview.mintlify.app/docs/astro/ide-deploy-git.md): Deploy code from the Astro IDE to Git - [Import CLI projects to the Astro IDE](https://astronomer-preview.mintlify.app/docs/astro/ide-import-cli-project.md): Import an existing local Astro project into the Astro IDE using the CLI. - [Import a Git project to the Astro IDE](https://astronomer-preview.mintlify.app/docs/astro/ide-import-git-project.md): Import an existing Git-based project from any provider into the Astro IDE. - [Import GitHub projects to the Astro IDE](https://astronomer-preview.mintlify.app/docs/astro/ide-import-github-project.md): Import an existing project into the Astro IDE by connecting your GitHub repository. - [Astro IDE overview](https://astronomer-preview.mintlify.app/docs/astro/ide-overview.md): Learn about the Astro IDE, a browser-based workspace for developing, testing, and deploying Apache Airflow Dags without a local development environment. - [Astro IDE Quickstart](https://astronomer-preview.mintlify.app/docs/astro/ide-quickstart.md): Astro IDE Quickstart - [Test and run code with the Astro IDE](https://astronomer-preview.mintlify.app/docs/astro/ide-test-run.md): Test and run code with the Astro IDE - [Import and export Airflow connections and variables](https://astronomer-preview.mintlify.app/docs/astro/import-export-connections-variables.md): Learn how to import and export Apache Airflow objects between Airflow environments on Astro - [Install Astro from the Azure Marketplace as an Azure Native ISV service](https://astronomer-preview.mintlify.app/docs/astro/install-azure.md): Learn how to install Astro from the Azure marketplace, which is recommended for all Azure-based teams. - [Set up IP Access List on Astro](https://astronomer-preview.mintlify.app/docs/astro/ip-access-list.md): Configure IP Access list to control the IP addresses from where your users can log in to Astro. - [Run images from Amazon Elastic Container Registry (ECR)](https://astronomer-preview.mintlify.app/docs/astro/kpo-ecr.md): Configure KPO to pull container images from Amazon ECR. - [Run images from Google Artifact Registry](https://astronomer-preview.mintlify.app/docs/astro/kpo-google-artifact-registry.md): Configure KPO to pull images from Google Artifact Registry. - [Mount temporary directory](https://astronomer-preview.mintlify.app/docs/astro/kpo-mount-temporary-directory.md): Learn how to mount a temporary directory for the KubernetesPodOperator on Astro. - [Run images from a private registry](https://astronomer-preview.mintlify.app/docs/astro/kpo-private-registry.md): Configure KPO to pull privately hosted container images. - [Run your Deployment's current Airflow image](https://astronomer-preview.mintlify.app/docs/astro/kpo-run-current-image.md): Configure KPO to run your Deployment's current Airflow image. - [Use secret environment variables](https://astronomer-preview.mintlify.app/docs/astro/kpo-secret-env-var.md): Use secret environment variables in tasks running on the Kubernetes executor. - [Use the @task.kubernetes decorator](https://astronomer-preview.mintlify.app/docs/astro/kpo-task-decorator.md): Learn how to run the task.kubernetes decorator on Astro. - [Configure task-level Pod resources](https://astronomer-preview.mintlify.app/docs/astro/kpo-task-level-resources.md): Learn how to configure task-level Pod resources for the KubernetesPodOperator on Astro. - [Configure tasks to run with the Kubernetes executor](https://astronomer-preview.mintlify.app/docs/astro/kubernetes-executor.md): Learn how to configure the Pods that the Kubernetes executor runs your tasks in. - [Run the KubernetesPodOperator on Astro](https://astronomer-preview.mintlify.app/docs/astro/kubernetespodoperator.md): Learn how to run the KubernetesPodOperator on Astro. This operator dynamically launches a Pod in Kubernetes for each task and terminates each Pod when the task is complete. - [Launch a Pod in an EKS cluster on AWS](https://astronomer-preview.mintlify.app/docs/astro/launch-pod-external-cluster-aws.md): Use the KubernetesPodOperator to launch Pods in an external EKS cluster on AWS from your Airflow instance. - [Launch a Pod in an AKS cluster on Azure](https://astronomer-preview.mintlify.app/docs/astro/launch-pod-external-cluster-azure.md): Use the KubernetesPodOperator to launch Pods in an external AKS cluster on Azure from your Airflow instance. - [Lineage](https://astronomer-preview.mintlify.app/docs/astro/lineage-graph.md): View lineage graph in Astro Observe. - [Log in to Astro](https://astronomer-preview.mintlify.app/docs/astro/log-in-to-astro.md): Learn how you can use the Cloud user interface (UI) and the Astro command-line interface (CLI) to log in to Astro. - [Manage Astro billing](https://astronomer-preview.mintlify.app/docs/astro/manage-billing.md): Change your billing details and view your current spend from the Astro UI. - [Manage Airflow connections, variables, and environment variables](https://astronomer-preview.mintlify.app/docs/astro/manage-connections-variables.md): Learn about different strategies for managing Airflow connections, variables, and environment variables in local environments and on Astro - [Manage dag runs](https://astronomer-preview.mintlify.app/docs/astro/manage-dags.md): Manage dag runs - [Manage Deployments programmatically using Deployment files](https://astronomer-preview.mintlify.app/docs/astro/manage-deployments-as-code.md): Manage an Astro Deployment using a Deployment file in YAML or JSON format - [Create and manage domains for your Organization](https://astronomer-preview.mintlify.app/docs/astro/manage-domains.md): Create and map domains to single sign-on platforms on Astro. - [Manage environment variables on Astro](https://astronomer-preview.mintlify.app/docs/astro/manage-env-vars.md): Learn how to manage environment variables on Astro - [Manage users in an Astro Organization](https://astronomer-preview.mintlify.app/docs/astro/manage-organization-users.md): Add, edit, or remove users within an Organization on Astro. - [Configure Teams on Astro](https://astronomer-preview.mintlify.app/docs/astro/manage-teams.md): Create, delete, and update Teams on Astro. - [Manage users in your Astro Workspace](https://astronomer-preview.mintlify.app/docs/astro/manage-workspace-users.md): Add, edit, or remove users within a Workspace on Astro. - [Configure Workspaces on Astro](https://astronomer-preview.mintlify.app/docs/astro/manage-workspaces.md): Create, delete, and update Workspaces on Astro. - [Migrate to Astro from Google Cloud Composer](https://astronomer-preview.mintlify.app/docs/astro/migrate-gcc.md): Get started on Astro by migrating your Airflow code from Google Cloud Composer (GCC). - [Migrate to Astro from Amazon MWAA](https://astronomer-preview.mintlify.app/docs/astro/migrate-mwaa.md): Get started on Astro Hosted by migrating your Airflow code from Amazon Managed Workflows for Apache Airflow (MWAA). - [Create network connections between Astro and external resources](https://astronomer-preview.mintlify.app/docs/astro/networking-overview.md): Learn about the fundamentals of creating network connections from Astro. - [Data quality](https://astronomer-preview.mintlify.app/docs/astro/observe-data-quality.md): Monitor tables and columns for volume, schema changes, and null percentages. View an overall data quality dashboard and investigate triggered issues. - [Get started with Observe](https://astronomer-preview.mintlify.app/docs/astro/observe-get-started.md): High-level technical prerequisites and setup steps for Astro Observe. - [Monitors in Astro Observe](https://astronomer-preview.mintlify.app/docs/astro/observe-monitors.md): Monitor data product and pipeline health in Astro Observe. - [Configure OpenLineage on Astro](https://astronomer-preview.mintlify.app/docs/astro/observe-openlineage.md): How Astro automatically configures OpenLineage and how to add additional backends. - [Service Level Agreements (SLAs)](https://astronomer-preview.mintlify.app/docs/astro/observe-slas.md): Create and manage SLAs in Astro Observe. - [Use TransformTransport to customize OpenLineage events](https://astronomer-preview.mintlify.app/docs/astro/observe-transformtransport.md): Use TransformTransport to customize OpenLineage events - [Export Astro reporting data](https://astronomer-preview.mintlify.app/docs/astro/org-dash-exports.md): Export data from Organization dashboards or configure conditional exports as alerts. - [Create and manage Organization API tokens](https://astronomer-preview.mintlify.app/docs/astro/organization-api-tokens.md): Create and manage Organization API tokens to automate key actions across all Workspaces in your Organization, like adding users and creating Deployments. - [View Organization dashboards](https://astronomer-preview.mintlify.app/docs/astro/organization-dashboard.md): View information about your Organization, Deployments, dags, and costs. - [Update settings for your Astro Organization](https://astronomer-preview.mintlify.app/docs/astro/organization-settings.md): Update high-level settings for an Astro Organization that apply to all Workspaces, Deployments, and users. - [Astro Documentation](https://astronomer-preview.mintlify.app/docs/astro/overview.md): Everything you need to know about Astronomer’s modern data orchestration tool for the cloud, powered by Apache Airflow. - [Global environment variables](https://astronomer-preview.mintlify.app/docs/astro/platform-variables.md): A list of environment variables that are set globally on Astro and should not be modified. - [Enable Private Network Egress](https://astronomer-preview.mintlify.app/docs/astro/private-network-egress.md): Enable Private Network Egress - [Astro release notes](https://astronomer-preview.mintlify.app/docs/astro/release-notes.md): Latest Astro product updates, features, and bug fixes. - [Subscribe to release notes](https://astronomer-preview.mintlify.app/docs/astro/release-notes-subscribe.md): Subscribe to Astronomer Release Notes RSS feeds, or have updates sent to your email or Slack account. - [Configure AWS PrivateLink for Remote Execution Agents](https://astronomer-preview.mintlify.app/docs/astro/remote-agents-aws-privatelink.md): Configure private connectivity between Remote Execution Agents and the Astro orchestration plane using AWS PrivateLink. - [Configure Azure Private Link for Remote Execution Agents](https://astronomer-preview.mintlify.app/docs/astro/remote-agents-azure-privatelink.md): Configure private connectivity between Remote Execution Agents and the Astro orchestration plane using Azure Private Link. - [Remote Execution Agent failure and recovery scenarios](https://astronomer-preview.mintlify.app/docs/astro/remote-agents-failure-scenarios.md): Remote Execution Agent failure and recovery scenarios - [Helm chart configuration reference](https://astronomer-preview.mintlify.app/docs/astro/remote-agents-helm-reference.md): Helm chart configuration reference - [Configure LoggingSidecar in a Remote Execution Agent](https://astronomer-preview.mintlify.app/docs/astro/remote-agents-logging-sidecar.md): Configure LoggingSidecar in a Remote Execution Agent - [Install Remote Execution Agents in a restricted kubernetes namespace](https://astronomer-preview.mintlify.app/docs/astro/remote-agents-restricted-kubernetes.md): Install Remote Execution Agents in a restricted kubernetes namespace - [Sentinel for Remote Execution Agents](https://astronomer-preview.mintlify.app/docs/astro/remote-agents-sentinel.md): Overview of the Sentinel service for Astro Remote Execution Agents. - [Register and configure agents](https://astronomer-preview.mintlify.app/docs/astro/remote-execution-configure-agents.md): Register and configure agents - [Configure DAG sources](https://astronomer-preview.mintlify.app/docs/astro/remote-execution-configure-dag-sources.md): Configure DAG sources - [Configure OpenLineage for a Remote Execution Agent](https://astronomer-preview.mintlify.app/docs/astro/remote-execution-configure-openlineage.md): Configure OpenLineage for a Remote Execution Agent - [Configure secrets backend](https://astronomer-preview.mintlify.app/docs/astro/remote-execution-configure-secrets-backend.md): Configure secrets backend - [Configure XCom backend](https://astronomer-preview.mintlify.app/docs/astro/remote-execution-configure-xcom-backend.md): Configure XCom backend - [Set up custom timetable support for Remote Execution](https://astronomer-preview.mintlify.app/docs/astro/remote-execution-custom-timetables.md): Configure the scheduler to access your secrets backend when using custom timetables on Remote Execution. - [Deploy a dbt project to Remote Execution Agents](https://astronomer-preview.mintlify.app/docs/astro/remote-execution-deploy-dbt.md): Learn how to make dbt project files available to Remote Execution Agents so that Cosmos can orchestrate dbt models as Airflow tasks. - [Get started with Remote Execution](https://astronomer-preview.mintlify.app/docs/astro/remote-execution-get-started.md): Get started with Remote Execution - [Configure logging](https://astronomer-preview.mintlify.app/docs/astro/remote-execution-logging-overview.md): Configure logging - [Remote Execution overview](https://astronomer-preview.mintlify.app/docs/astro/remote-execution-overview.md): Remote Execution overview - [Remote Execution shared responsibility model](https://astronomer-preview.mintlify.app/docs/astro/remote-execution-shared-responsibility.md): Remote Execution shared responsibility model - [Show Remote Execution Agent task logs in Airflow UI](https://astronomer-preview.mintlify.app/docs/astro/remote-task-logs-af-ui.md): Show Remote Execution Agent task logs in Airflow UI - [Send Remote Execution Agent task logs to external logging provider](https://astronomer-preview.mintlify.app/docs/astro/remote-task-logs-external.md): Send Remote Execution Agent task logs to external logging provider - [Resilience](https://astronomer-preview.mintlify.app/docs/astro/resilience.md): Learn how Astronomer leverages Availability Zones to make the control plane and data plane resilient. - [Astro Hosted resource reference](https://astronomer-preview.mintlify.app/docs/astro/resource-reference-hosted.md): Reference of all supported infrastructure for new Astro Hosted clusters. - [Root cause analysis and AI log summaries](https://astronomer-preview.mintlify.app/docs/astro/root-cause-analysis.md): Use root cause analysis and AI log summaries to diagnose failures in Astro Observe. - [Run your first dag on Astro](https://astronomer-preview.mintlify.app/docs/astro/run-first-dag.md): Run your first dag on Astro - [Configure a secrets backend](https://astronomer-preview.mintlify.app/docs/astro/secrets-backend.md): Learn to configure a secrets backend on Astro to store Airflow connections and variables - [Set up AWS Systems Manager (SSM) Parameter Store](https://astronomer-preview.mintlify.app/docs/astro/secrets-backend/aws-paramstore.md): Set up AWS Systems Manager (SSM) Parameter Store - [Set up AWS Secrets Manager as your secrets backend](https://astronomer-preview.mintlify.app/docs/astro/secrets-backend/aws-secretsmanager.md): Set up AWS Secrets Manager as your secrets backend - [Set up Azure Key Vault as your secrets backend](https://astronomer-preview.mintlify.app/docs/astro/secrets-backend/azure-key-vault.md): Set up Azure Key Vault as your secrets backend - [Set up Google Cloud Secret Manager as your secrets backend](https://astronomer-preview.mintlify.app/docs/astro/secrets-backend/gcp-secretsmanager.md): Set up Google Cloud Secret Manager as your secrets backend - [Set up Hashicorp Vault as your secrets backend](https://astronomer-preview.mintlify.app/docs/astro/secrets-backend/hashicorp-vault.md): Set up Hashicorp Vault as your secrets backend - [Secrets management](https://astronomer-preview.mintlify.app/docs/astro/secrets-management.md): Learn how Astronomer secures your sensitive information and supports secrets management integration - [Security in Astro](https://astronomer-preview.mintlify.app/docs/astro/security.md): Learn how Astro responds to and implements a variety of security concepts - [Develop a CI/CD workflow for deploying code to Astro](https://astronomer-preview.mintlify.app/docs/astro/set-up-ci-cd.md): Develop a CI/CD workflow for deploying code to Astro - [Set up SCIM provisioning on Astro](https://astronomer-preview.mintlify.app/docs/astro/set-up-scim-provisioning.md): Configure SCIM provisioning to import groups of users from your identity provider to Astro as Teams. - [Shared responsibility model](https://astronomer-preview.mintlify.app/docs/astro/shared-responsibility-model.md): Astronomer's policy on shared responsibilities between our team and our customers. - [Subscribe to Astro from the AWS Marketplace](https://astronomer-preview.mintlify.app/docs/astro/subscribe-aws.md): Learn how to subscribe to Astro from the AWS marketplace. - [Astro Terraform Provider](https://astronomer-preview.mintlify.app/docs/astro/terraform-provider.md): Learn about the Astro Terraform Provider and how to work with it. - [Transfer a Deployment to a different Workspace](https://astronomer-preview.mintlify.app/docs/astro/transfer-a-deployment.md): Transfer a Deployment to another Workspace in the same cluster. - [Start your Astro trial](https://astronomer-preview.mintlify.app/docs/astro/trial.md): Start your Astro trial - [Astro user permissions reference](https://astronomer-preview.mintlify.app/docs/astro/user-permissions.md): Learn about Astronomer's RBAC system and how to assign roles to users. - [View clusters](https://astronomer-preview.mintlify.app/docs/astro/view-clusters.md): View details about your clusters and cluster-specific configurations. - [View Airflow component and task logs for a Deployment](https://astronomer-preview.mintlify.app/docs/astro/view-logs.md): View logs for your Deployments both locally and on Astro. - [Create and manage Workspace API tokens](https://astronomer-preview.mintlify.app/docs/astro/workspace-api-tokens.md): Create and manage Workspace API tokens to automate key Workspace actions, like adding users and creating Deployments. - [Airflow cluster policies](https://astronomer-preview.mintlify.app/docs/learn/2.x/airflow-advanced-cluster-policies.md): Learn about everything you need to use the Apache Airflow cluster policies. - [Orchestrate Ray jobs on Anyscale with Apache Airflow®](https://astronomer-preview.mintlify.app/docs/learn/2.x/airflow-anyscale.md): Learn how to use the Anyscale provider package to orchestrate Ray jobs on Anyscale with Apache Airflow®. - [Run a task in Azure Container Instances with Airflow](https://astronomer-preview.mintlify.app/docs/learn/2.x/airflow-azure-container-instances.md): Learn how to orchestrate containers with Azure Container Instances from your Airflow DAGs. - [Run Azure Data Factory pipelines with Airflow](https://astronomer-preview.mintlify.app/docs/learn/2.x/airflow-azure-data-factory-integration.md): Learn how to orchestrate remote jobs in Azure Data Factory with your Apache Airflow DAGs. - [Branching in Airflow](https://astronomer-preview.mintlify.app/docs/learn/2.x/airflow-branch-operator.md): Learn about Airflow’s multiple options for building conditional logic and branching within DAGs, including the BranchPythonOperator and ShortCircuitOperator. - [Orchestrate Cohere LLMs with Apache Airflow](https://astronomer-preview.mintlify.app/docs/learn/2.x/airflow-cohere.md): Learn how to integrate Cohere and Airflow. - [Apache Airflow® components](https://astronomer-preview.mintlify.app/docs/learn/2.x/airflow-components.md): Understand the core components of Apache Airflow®. Review their functions and find out which components to run for specific use cases. - [Access the Apache Airflow context](https://astronomer-preview.mintlify.app/docs/learn/2.x/airflow-context.md): Access the Airflow context in your tasks. - [DAG-level parameters in Airflow](https://astronomer-preview.mintlify.app/docs/learn/2.x/airflow-dag-parameters.md): Learn about all important DAG-level parameters in Airflow. - [Understanding the Airflow metadata database](https://astronomer-preview.mintlify.app/docs/learn/2.x/airflow-database.md): Learn about everything you need to use the Apache Airflow metadata database. - [Orchestrate Databricks jobs with Airflow](https://astronomer-preview.mintlify.app/docs/learn/2.x/airflow-databricks.md): Orchestrate Databricks jobs with your Airflow DAGs. - [Datasets and data-aware scheduling in Airflow](https://astronomer-preview.mintlify.app/docs/learn/2.x/airflow-datasets.md): Using datasets to implement DAG dependencies and scheduling in Airflow. - [Orchestrate dbt Core jobs with Airflow and Cosmos](https://astronomer-preview.mintlify.app/docs/learn/2.x/airflow-dbt.md): Orchestrate dbt Core jobs with Airflow and Cosmos - [Orchestrate dbt Cloud jobs with Airflow](https://astronomer-preview.mintlify.app/docs/learn/2.x/airflow-dbt-cloud.md): Learn how to use the dbt Cloud Provider to orchestrate dbt Cloud jobs with Airflow. - [Introduction to the TaskFlow API and Airflow decorators](https://astronomer-preview.mintlify.app/docs/learn/2.x/airflow-decorators.md): An overview of Airflow decorators and how they can improve the DAG authoring experience. - [Use DuckDB with Apache Airflow](https://astronomer-preview.mintlify.app/docs/learn/2.x/airflow-duckdb.md): Learn how to use DuckDB with Airflow. - [Apache Airflow® Executors](https://astronomer-preview.mintlify.app/docs/learn/2.x/airflow-executors-explained.md): An introduction to Apache Airflow® Executors - [Use Fivetran with Apache Airflow](https://astronomer-preview.mintlify.app/docs/learn/2.x/airflow-fivetran.md): Learn how to orchestrate Fivetran syncs using Airflow - [Orchestrate Great Expectations with Airflow](https://astronomer-preview.mintlify.app/docs/learn/2.x/airflow-great-expectations.md): Orchestrate Great Expectations data quality checks with your Airflow DAGs. - [Custom hooks and operators](https://astronomer-preview.mintlify.app/docs/learn/2.x/airflow-importing-custom-hooks-operators.md): How to correctly import custom hooks and operators. - [Run tasks in an isolated environment in Apache Airflow](https://astronomer-preview.mintlify.app/docs/learn/2.x/airflow-isolated-environments.md): Learn how to run an Airflow task in an isolated environment. - [Use Apache Kafka with Apache Airflow](https://astronomer-preview.mintlify.app/docs/learn/2.x/airflow-kafka.md): How to produce to and consume from Kafka topics using the Kafka Airflow provider - [Use a listener to send a Slack notification when a dataset is updated](https://astronomer-preview.mintlify.app/docs/learn/2.x/airflow-listeners.md): Learn how to use Airflow listeners. - [Use MLflow with Apache Airflow](https://astronomer-preview.mintlify.app/docs/learn/2.x/airflow-mlflow.md): How to use MLflow with Airflow in three different ways. - [Best practices for orchestrating MLOps pipelines with Airflow](https://astronomer-preview.mintlify.app/docs/learn/2.x/airflow-mlops.md): Learn how to use Airflow to run machine learning in production. - [Orchestrate MongoDB operations with Apache Airflow](https://astronomer-preview.mintlify.app/docs/learn/2.x/airflow-mongodb.md): Learn how to load vector embeddings into MongoDB with Apache Airflow. - [Use Airflow object storage to interact with cloud storage in an ML pipeline](https://astronomer-preview.mintlify.app/docs/learn/2.x/airflow-object-storage-tutorial.md): Learn how to use Airflow object storage. - [Orchestrate OpenAI operations with Apache Airflow](https://astronomer-preview.mintlify.app/docs/learn/2.x/airflow-openai.md): Learn how to integrate OpenAI and Airflow. - [Integrate OpenLineage and Airflow](https://astronomer-preview.mintlify.app/docs/learn/2.x/airflow-openlineage.md): Learn about OpenLineage concepts and benefits of integrating with Airflow. - [Orchestrate OpenSearch operations with Apache Airflow](https://astronomer-preview.mintlify.app/docs/learn/2.x/airflow-opensearch.md): Learn how to integrate OpenSearch and Airflow. - [Create and use params in Airflow](https://astronomer-preview.mintlify.app/docs/learn/2.x/airflow-params.md): Create and use DAG and task-level params in Airflow. - [Pass data between tasks](https://astronomer-preview.mintlify.app/docs/learn/2.x/airflow-passing-data-between-tasks.md): Learn more about the most common methods to implement data sharing between your Airflow tasks, including an in-depth explanation of XCom. - [Orchestrate pgvector operations with Apache Airflow](https://astronomer-preview.mintlify.app/docs/learn/2.x/airflow-pgvector.md): Learn how to integrate pgvector and Airflow. - [Orchestrate Pinecone operations with Apache Airflow](https://astronomer-preview.mintlify.app/docs/learn/2.x/airflow-pinecone.md): Learn how to integrate Pinecone and Airflow. - [Airflow pools](https://astronomer-preview.mintlify.app/docs/learn/2.x/airflow-pools.md): Use pools to control Airflow task parallelism. - [Orchestrate semantic querying in Qdrant with Airflow](https://astronomer-preview.mintlify.app/docs/learn/2.x/airflow-qdrant.md): Learn how to integrate Qdrant and Airflow. - [Apache Airflow® Quickstart](https://astronomer-preview.mintlify.app/docs/learn/2.x/airflow-quickstart.md): Apache Airflow® Quickstart - [Orchestrate Ray jobs with Apache Airflow®](https://astronomer-preview.mintlify.app/docs/learn/2.x/airflow-ray.md): Learn how to use the Ray provider package to orchestrate Ray jobs with Apache Airflow®. - [Orchestrate Redshift operations with Airflow](https://astronomer-preview.mintlify.app/docs/learn/2.x/airflow-redshift.md): Orchestrate Redshift queries from your Airflow DAGs. - [Train a machine learning model with SageMaker and Airflow](https://astronomer-preview.mintlify.app/docs/learn/2.x/airflow-sagemaker.md): Follow a step-by-step tutorial for using Airflow to orchestrate the training and testing of a SageMaker model. - [Scaling Airflow to optimize performance](https://astronomer-preview.mintlify.app/docs/learn/2.x/airflow-scaling-workers.md): See which parameters to modify when scaling up data pipelines to make the most of Airflow. Learn about the environment, DAG, and task-level settings. - [Use setup and teardown tasks in Airflow](https://astronomer-preview.mintlify.app/docs/learn/2.x/airflow-setup-teardown.md): Learn how to use setup and teardown tasks to manage task resources in Airflow. - [Orchestrate Snowflake Queries with Airflow](https://astronomer-preview.mintlify.app/docs/learn/2.x/airflow-snowflake.md): Get enhanced observability and compute savings while orchestrating Snowflake jobs from your Airflow DAGs. - [Orchestrate Snowpark Machine Learning Workflows with Apache Airflow](https://astronomer-preview.mintlify.app/docs/learn/2.x/airflow-snowpark.md): Learn how to integrate Snowpark and Airflow. - [Using Airflow to Execute SQL](https://astronomer-preview.mintlify.app/docs/learn/2.x/airflow-sql.md): Learn the best practices for executing SQL from your DAG. Get to know Airflow’s SQL-related operators and see how to use Airflow for common SQL use cases. - [Run data quality checks using SQL check operators](https://astronomer-preview.mintlify.app/docs/learn/2.x/airflow-sql-data-quality.md): Learn how to use the QLColumnCheckOperator, SQLTableCheckOperator and SQLCheckOperator. - [Airflow trigger rules](https://astronomer-preview.mintlify.app/docs/learn/2.x/airflow-trigger-rules.md): Learn about available trigger rules and how to use them. - [An introduction to the Airflow UI](https://astronomer-preview.mintlify.app/docs/learn/2.x/airflow-ui.md): Explore the Airflow UI, which helps you monitor and troubleshoot your data pipelines. Learn about some of its key features and visualizations. - [Use Airflow variables](https://astronomer-preview.mintlify.app/docs/learn/2.x/airflow-variables.md): Create and use Airflow variables. - [Orchestrate Weaviate operations with Apache Airflow](https://astronomer-preview.mintlify.app/docs/learn/2.x/airflow-weaviate.md): Learn how to integrate Weaviate and Airflow. - [Manage your ML models with Weights and Biases and Airflow](https://astronomer-preview.mintlify.app/docs/learn/2.x/airflow-weights-and-biases.md): Learn how to use Airflow and Weights and Biases to manage and visualize your ML model lifecycle. - [Get started with Astro Observe](https://astronomer-preview.mintlify.app/docs/learn/2.x/astro-observe-quickstart.md): Learn how to get started with Astro Observe - [Using the BashOperator](https://astronomer-preview.mintlify.app/docs/learn/2.x/bashoperator.md): Learn how to use the BashOperator to run bash commands and bash scripts. Review examples of how to run scripts in other languages than Python. - [Clean up the Airflow metadata database using DAGs](https://astronomer-preview.mintlify.app/docs/learn/2.x/cleanup-dag-tutorial.md): Learn how to remove unnecessary data from the Airflow metadata database using the `airflow db clean` command from a DAG. - [ML pipelines with the Astro Cloud IDE](https://astronomer-preview.mintlify.app/docs/learn/2.x/cloud-ide-tutorial.md): Use tutorials and guides to make the most out of Airflow and Astronomer. - [Manage connections in Apache Airflow](https://astronomer-preview.mintlify.app/docs/learn/2.x/connections.md): Learn how to set up, manage, and maintain different types of connections in Apache Airflow. Use example connection configurations as the basis for your own connections. - [Create an Azure Blob Storage connection in Airflow](https://astronomer-preview.mintlify.app/docs/learn/2.x/connections/azure-blob-storage.md): Learn how to create an Azure Blob Storage connection in Airflow. - [Create an Azure Data Factory connection in Airflow](https://astronomer-preview.mintlify.app/docs/learn/2.x/connections/azure-data-factory.md): Learn how to create an Azure Data Factory connection in Airflow. - [Create a BigQuery connection in Airflow](https://astronomer-preview.mintlify.app/docs/learn/2.x/connections/bigquery.md): Learn how to create a BigQuery connection in Airflow. - [Create a Databricks connection in Airflow](https://astronomer-preview.mintlify.app/docs/learn/2.x/connections/databricks.md): Learn how to create a Databricks connection in Airflow. - [Create a dbt Cloud connection in Airflow](https://astronomer-preview.mintlify.app/docs/learn/2.x/connections/dbt-cloud.md): Learn how to create a dbt Cloud connection in Airflow. - [Create a Microsoft Entra Workload ID connection in Airflow](https://astronomer-preview.mintlify.app/docs/learn/2.x/connections/entra-workload-identity.md): Learn how to create an Azure Workload Identity connection in Airflow. - [Create a Microsoft SQL Server connection in Airflow](https://astronomer-preview.mintlify.app/docs/learn/2.x/connections/ms-sqlserver.md): Learn how to create a Microsoft SQL Server connection in Airflow. - [Create a Postgres connection in Airflow](https://astronomer-preview.mintlify.app/docs/learn/2.x/connections/postgres.md): Learn how to create a Postgres connection in Airflow. - [Create an Amazon Redshift Connection in Airflow](https://astronomer-preview.mintlify.app/docs/learn/2.x/connections/redshift.md): Learn how to create an Amazon Redshift connection in Airflow. - [Create a Snowflake Connection in Airflow](https://astronomer-preview.mintlify.app/docs/learn/2.x/connections/snowflake.md): Learn how to create a Snowflake connection in Airflow. - [Cross-DAG dependencies](https://astronomer-preview.mintlify.app/docs/learn/2.x/cross-dag-dependencies.md): How to implement dependencies between your Airflow DAGs. - [Create DAG documentation in Apache Airflow](https://astronomer-preview.mintlify.app/docs/learn/2.x/custom-airflow-ui-docs-tutorial.md): Use Apache Airflow's built-in documentation features to generate documentation for your DAGs in the Airflow UI. - [Strategies for custom XCom backends in Airflow](https://astronomer-preview.mintlify.app/docs/learn/2.x/custom-xcom-backend-strategies.md): Use this guide to learn about different ways you can set up custom XCom backends. - [Set up a custom XCom backend using object storage](https://astronomer-preview.mintlify.app/docs/learn/2.x/custom-xcom-backends-tutorial.md): Use this tutorial to learn how to set up a custom XCom backend with object storage. - [DAG writing best practices in Apache Airflow](https://astronomer-preview.mintlify.app/docs/learn/2.x/dag-best-practices.md): Keep up to date with the best practices for developing efficient, secure, and scalable DAGs using Airflow. Learn about DAG design and data orchestration. - [Use DAG Factory to create DAGs](https://astronomer-preview.mintlify.app/docs/learn/2.x/dag-factory.md): Learn how to dynamically convert YAML files into Apache Airflow® DAGs with the DAG Factory, an open source project that makes creating DAGs easy. - [Introduction to Apache Airflow® DAGs](https://astronomer-preview.mintlify.app/docs/learn/2.x/dags.md): Learn how to write DAGs and get tips on how to define an Apache Airflow® DAG in Python. Learn all about DAG parameters and their settings. - [Leveraging data products for health and performance benefits](https://astronomer-preview.mintlify.app/docs/learn/2.x/data-products.md): Learn about the health and performance monitoring benefits of data products. - [Data quality and Airflow](https://astronomer-preview.mintlify.app/docs/learn/2.x/data-quality.md): Check the quality of your data using Airflow. - [Debug DAGs](https://astronomer-preview.mintlify.app/docs/learn/2.x/debugging-dags.md): Troubleshoot Airflow DAGs - [Deferrable operators](https://astronomer-preview.mintlify.app/docs/learn/2.x/deferrable-operators.md): Implement deferrable operators to save cost and resources with Airflow. - [Create dynamic Airflow tasks](https://astronomer-preview.mintlify.app/docs/learn/2.x/dynamic-tasks.md): How to dynamically create tasks at runtime in your Airflow DAGs. - [Dynamically generate DAGs in Airflow](https://astronomer-preview.mintlify.app/docs/learn/2.x/dynamically-generating-dags.md): Get to know the best ways to dynamically generate DAGs in Apache Airflow. Use examples to generate DAGs using single- and multiple-file methods. - [Manage Apache Airflow® DAG notifications](https://astronomer-preview.mintlify.app/docs/learn/2.x/error-notifications-in-airflow.md): Master the basics of Apache Airflow® notifications. Learn how to set up automatic email and Slack notifications to be alerted of events in your DAGs. - [Create a data-aware and dynamic ETL pipeline](https://astronomer-preview.mintlify.app/docs/learn/2.x/etl-datasets.md): Create a data-aware and dynamic ETL pipeline - [Microsoft Teams notifications](https://astronomer-preview.mintlify.app/docs/learn/2.x/example-ms-teams-callback.md): Configure notifications in Microsoft teams for DAG runs and tasks using Airflow callbacks. - [Execute a Jupyter notebook with Airflow](https://astronomer-preview.mintlify.app/docs/learn/2.x/execute-notebooks.md): Run a parameterized Jupyter notebook using Airflow and the Astro CLI. - [Get started with Apache Airflow, Part 1: Write and run your first DAG](https://astronomer-preview.mintlify.app/docs/learn/2.x/get-started-with-airflow.md): Use tutorials and guides to make the most out of Airflow and Astronomer. - [Get started with Apache Airflow, Part 2: Providers, connections, and variables](https://astronomer-preview.mintlify.app/docs/learn/2.x/get-started-with-airflow-part-2.md): Learn the core Apache Airflow concepts of using providers and connections. - [Glossary](https://astronomer-preview.mintlify.app/docs/learn/2.x/glossary.md): A list of key Apache Airflow, observability, and other data terms and definitions that can help you learn important concepts. - [An introduction to Apache Airflow®](https://astronomer-preview.mintlify.app/docs/learn/2.x/intro-to-airflow.md): Learn what Apache Airflow® is and what problems it solves. Get free access to valuable learning resources. - [Use the KubernetesPodOperator](https://astronomer-preview.mintlify.app/docs/learn/2.x/kubepod-operator.md): Use the KubernetesPodOperator in Airflow to run tasks in Kubernetes Pods - [Airflow logging](https://astronomer-preview.mintlify.app/docs/learn/2.x/logging.md): An introduction to Airflow logging. - [Manage Airflow code](https://astronomer-preview.mintlify.app/docs/learn/2.x/managing-airflow-code.md): Learn best practices for Airflow project organization, such as when to separate out DAGs into multiple projects and how to manage code used across different projects. - [Manage task and task group dependencies in Airflow](https://astronomer-preview.mintlify.app/docs/learn/2.x/managing-dependencies.md): Learn how to manage dependencies between tasks and TaskGroups in Apache Airflow, including how to set dynamic dependencies. - [Integrate OpenLineage and Airflow with Marquez](https://astronomer-preview.mintlify.app/docs/learn/2.x/marquez.md): Use OpenLineage and Marquez to get lineage metadata locally from your Airflow DAGs. - [Customize Operator Extra Links](https://astronomer-preview.mintlify.app/docs/learn/2.x/operator-extra-link-tutorial.md): Learn how to add operator extra links using the AirflowExtraLinkPlugin. - [Learn](https://astronomer-preview.mintlify.app/docs/learn/2.x/overview.md): Use tutorials and concepts to learn everything you need to know about Apache Airflow® and Astro - [Implementing proactive alerting](https://astronomer-preview.mintlify.app/docs/learn/2.x/proactive-alerting.md): Learn about implementing proactive alerting as part of your observability solution. - [Develop Airflow DAGs locally with PyCharm](https://astronomer-preview.mintlify.app/docs/learn/2.x/pycharm-local-dev.md): Integrate the Astro CLI with PyCharm for local development. - [ELT with BigQuery, dbt, and Apache Airflow® for eCommerce](https://astronomer-preview.mintlify.app/docs/learn/2.x/reference-architecture-elt-bigquery-dbt.md): Learn how to build an end-to-end ELT pipeline with Apache Airflow®, BigQuery, and dbt Core. - [ELT with Snowflake and Apache Airflow® for eCommerce](https://astronomer-preview.mintlify.app/docs/learn/2.x/reference-architecture-elt-snowflake.md): Learn how to build an end-to-end ELT pipeline with Apache Airflow® and Snowflake. - [Processing User Feedback: an LLM-fine-tuning reference architecture with Ray on Anyscale](https://astronomer-preview.mintlify.app/docs/learn/2.x/reference-architecture-fine-tuning-anyscale.md): Learn how to fine-tune an LLM to process and categorize user feedback with Airflow and Ray on Anyscale. - [Hybrid Search for eCommerce reference architecture](https://astronomer-preview.mintlify.app/docs/learn/2.x/reference-architecture-hybrid-search.md): Learn how to build a hybrid search application with Apache Airflow® and Weaviate. - [Batch inference for product insights with Apache Airflow®](https://astronomer-preview.mintlify.app/docs/learn/2.x/reference-architecture-product-insights.md): Learn how to build a batch inference pipeline with Apache Airflow® and OpenAI. - [SnowPatrol: Snowflake Usage Anomaly Detection & Alerting System](https://astronomer-preview.mintlify.app/docs/learn/2.x/reference-architecture-snowpatrol.md): Learn how to build an anomaly detection system for Snowflake with Apache Airflow®. - [Rerun Airflow DAGs and tasks](https://astronomer-preview.mintlify.app/docs/learn/2.x/rerunning-dags.md): How to configure retries, catchup, backfill, and clear task instances in Airflow. - [Schedule DAGs in Apache Airflow®](https://astronomer-preview.mintlify.app/docs/learn/2.x/scheduling-in-airflow.md): Get to know Airflow scheduling concepts and different ways to schedule a DAG. - [How to share code between multiple Airflow projects](https://astronomer-preview.mintlify.app/docs/learn/2.x/sharing-code-multiple-projects.md): A description of the various ways to reuse and share code between multiple projects, with pros and cons of each solution. - [Run Soda Core checks with Airflow](https://astronomer-preview.mintlify.app/docs/learn/2.x/soda-data-quality.md): Learn how to orchestrate Soda Core data quality checks with your Airflow DAGs. - [Airflow SubDAGs](https://astronomer-preview.mintlify.app/docs/learn/2.x/subdags.md): Use SubDAGs to create Airflow modular workflows. - [Airflow task groups](https://astronomer-preview.mintlify.app/docs/learn/2.x/task-groups.md): Follow Astronomer’s step-by-step guide to use task groups for organizing tasks within the grid view of the Airflow user interface. - [Use Airflow templates](https://astronomer-preview.mintlify.app/docs/learn/2.x/templating.md): Learn about Jinja templating in Apache Airflow and see examples of how to pass dynamic information into task instances at runtime. - [Test Airflow DAGs](https://astronomer-preview.mintlify.app/docs/learn/2.x/testing-airflow.md): Learn about testing Airflow DAGs and gain insight into various types of tests — validation testing, unit testing, and data and pipeline integrity testing. - [ELT with Apache Airflow® and Databricks](https://astronomer-preview.mintlify.app/docs/learn/2.x/use-case-airflow-databricks.md): Use Airflow to orchestrate data loading and transformation in Databricks. - [Orchestrate machine learning pipelines with Airflow datasets](https://astronomer-preview.mintlify.app/docs/learn/2.x/use-case-airflow-datasets-multi-team-ml.md): Using Airflow Datasets to create a Producer/Consumer ML pipeline in Airflow - [LLMOps: Automatic retrieval-augmented generation with Airflow, GPT-4 and Weaviate](https://astronomer-preview.mintlify.app/docs/learn/2.x/use-case-airflow-llm-rag-finance.md): Use Airflow and Weaviate in a RAG pipeline. - [Predict possum tail length using MLflow, Airflow, and linear regression](https://astronomer-preview.mintlify.app/docs/learn/2.x/use-case-airflow-mlflow.md): Use Airflow and MLflow to conduct and track a regression model. - [Run an integrated ELT and ML pipeline on Stripe data in Airflow](https://astronomer-preview.mintlify.app/docs/learn/2.x/use-case-elt-ml-finance.md): Use Airflow, the Astro Python SDK and Datasets in a data-driven way. - [Use Cohere and OpenSearch to analyze customer feedback in an MLOps pipeline](https://astronomer-preview.mintlify.app/docs/learn/2.x/use-case-llm-customer-feedback.md): Use Cohere and OpenSearch to analyze customer feedback in an MLOps pipeline. - [Use Airflow setup/ teardown to run data quality checks in an MLOps pipeline](https://astronomer-preview.mintlify.app/docs/learn/2.x/use-case-setup-teardown-data-quality.md): Use setup/ teardown to run data quality checks before running a classification model on roses. - [Airflow plugins](https://astronomer-preview.mintlify.app/docs/learn/2.x/using-airflow-plugins.md): How to use Airflow plugins. - [Leverage SLAs for enhanced data quality monitoring](https://astronomer-preview.mintlify.app/docs/learn/2.x/using-slas.md): Learn about use cases and best practices for SLAs. - [Develop Airflow DAGs locally with VS Code](https://astronomer-preview.mintlify.app/docs/learn/2.x/vscode-local-dev.md): Integrate the Astro CLI with VS Code for local Airflow development. - [Airflow hooks](https://astronomer-preview.mintlify.app/docs/learn/2.x/what-is-a-hook.md): Learn about hooks and how they should be used in Apache Airflow. See an example of implementing two different hooks in a DAG. - [Airflow sensors](https://astronomer-preview.mintlify.app/docs/learn/2.x/what-is-a-sensor.md): Get an overview of Airflow sensors and learn best practices for implementing sensors in production. - [Airflow operators](https://astronomer-preview.mintlify.app/docs/learn/2.x/what-is-an-operator.md): Learn the basics of operators, which are the building blocks of Airflow DAGs. See an example of how to implement several common operators available in Apache Airflow. - [Advanced asset-based scheduling in Apache Airflow®](https://astronomer-preview.mintlify.app/docs/learn/airflow-advanced-asset-scheduling.md): Using assets to implement DAG dependencies and scheduling in Airflow. - [Airflow cluster policies](https://astronomer-preview.mintlify.app/docs/learn/airflow-advanced-cluster-policies.md): Learn about everything you need to use the Apache Airflow cluster policies. - [Orchestrate Ray jobs on Anyscale with Apache Airflow®](https://astronomer-preview.mintlify.app/docs/learn/airflow-anyscale.md): Learn how to use the Anyscale provider package to orchestrate Ray jobs on Anyscale with Apache Airflow®. - [@asset syntax in Apache Airflow®](https://astronomer-preview.mintlify.app/docs/learn/airflow-asset-decorator.md): Using the @asset decorator to create a Dag with one task that updates an asset. - [Run a task in Azure Container Instances with Airflow](https://astronomer-preview.mintlify.app/docs/learn/airflow-azure-container-instances.md): Learn how to orchestrate containers with Azure Container Instances from your Airflow DAGs. - [Run Azure Data Factory pipelines with Airflow](https://astronomer-preview.mintlify.app/docs/learn/airflow-azure-data-factory-integration.md): Learn how to orchestrate remote jobs in Azure Data Factory with your Apache Airflow DAGs. - [Branching in Airflow](https://astronomer-preview.mintlify.app/docs/learn/airflow-branch-operator.md): Learn about Airflow's multiple options for building conditional logic and branching within DAGs, including the BranchPythonOperator and ShortCircuitOperator. - [Orchestrate Cohere LLMs with Apache Airflow](https://astronomer-preview.mintlify.app/docs/learn/airflow-cohere.md): Learn how to integrate Cohere and Airflow. - [Apache Airflow® components](https://astronomer-preview.mintlify.app/docs/learn/airflow-components.md): Understand the core components of Apache Airflow®. Review their functions and find out which components to run for specific use cases. - [Access the Apache Airflow context](https://astronomer-preview.mintlify.app/docs/learn/airflow-context.md): Access the Airflow context in your tasks. - [DAG-level parameters in Airflow](https://astronomer-preview.mintlify.app/docs/learn/airflow-dag-parameters.md): Learn about all important DAG-level parameters in Airflow. - [DAG Versioning and DAG Bundles](https://astronomer-preview.mintlify.app/docs/learn/airflow-dag-versioning.md): Learn how the DAG versioning feature functions and how to configure a versioned DAG bundle. - [Understanding the Airflow metadata database](https://astronomer-preview.mintlify.app/docs/learn/airflow-database.md): Learn about everything you need to use the Apache Airflow metadata database. - [Tutorial: How to Orchestrate Databricks Jobs with Airflow](https://astronomer-preview.mintlify.app/docs/learn/airflow-databricks.md): Step-by-step guide to orchestrating Databricks with Airflow. Learn to trigger notebooks, run jobs, and build data pipelines. Includes example Dag code. - [Basic asset-based scheduling in Apache Airflow®](https://astronomer-preview.mintlify.app/docs/learn/airflow-datasets.md): Using assets to schedule Dags based on successful completion of tasks. - [Orchestrate dbt Core projects with Airflow and Cosmos](https://astronomer-preview.mintlify.app/docs/learn/airflow-dbt.md): Learn how to use Cosmos to orchestrate dbt Core projects with Airflow. - [Orchestrate dbt Cloud jobs with Airflow](https://astronomer-preview.mintlify.app/docs/learn/airflow-dbt-cloud.md): Learn how to use the dbt Cloud Provider to orchestrate dbt Cloud jobs with Airflow. - [Introduction to the TaskFlow API and Airflow decorators](https://astronomer-preview.mintlify.app/docs/learn/airflow-decorators.md): An overview of Airflow decorators and how they can improve the DAG authoring experience. - [Use DuckDB with Apache Airflow](https://astronomer-preview.mintlify.app/docs/learn/airflow-duckdb.md): Learn how to use DuckDB with Airflow. - [Event-driven scheduling](https://astronomer-preview.mintlify.app/docs/learn/airflow-event-driven-scheduling.md): Learn how schedule DAGs based on messages in a message queue. - [Apache Airflow® Executors](https://astronomer-preview.mintlify.app/docs/learn/airflow-executors-explained.md): An introduction to Apache Airflow® Executors - [Use Fivetran with Apache Airflow](https://astronomer-preview.mintlify.app/docs/learn/airflow-fivetran.md): Learn how to orchestrate Fivetran syncs using Airflow - [Orchestrate Great Expectations with Airflow](https://astronomer-preview.mintlify.app/docs/learn/airflow-great-expectations.md): Orchestrate Great Expectations data quality checks with your Airflow Dags. - [Human-in-the-loop workflows with Airflow](https://astronomer-preview.mintlify.app/docs/learn/airflow-human-in-the-loop.md): Learn how include humans in your dags with the human-in-the-loop feature. - [Custom hooks and operators](https://astronomer-preview.mintlify.app/docs/learn/airflow-importing-custom-hooks-operators.md): How to correctly import custom hooks and operators. - [Run tasks in an isolated environment in Apache Airflow](https://astronomer-preview.mintlify.app/docs/learn/airflow-isolated-environments.md): Learn how to run an Airflow task in an isolated environment. - [Use Apache Kafka with Apache Airflow](https://astronomer-preview.mintlify.app/docs/learn/airflow-kafka.md): How to produce to and consume from Kafka topics using the Kafka Airflow provider - [Use a listener to send a Slack notification when a dataset is updated](https://astronomer-preview.mintlify.app/docs/learn/airflow-listeners.md): Learn how to use Airflow listeners. - [Best practices for orchestrating MLOps pipelines with Airflow](https://astronomer-preview.mintlify.app/docs/learn/airflow-mlops.md): Learn how to use Airflow to run machine learning in production. - [Orchestrate MongoDB operations with Apache Airflow](https://astronomer-preview.mintlify.app/docs/learn/airflow-mongodb.md): Learn how to load vector embeddings into MongoDB with Apache Airflow. - [Run Airflow tasks in other languages](https://astronomer-preview.mintlify.app/docs/learn/airflow-multilanguage.md): Learn how to run Airflow tasks in languages other than Python. - [Use Airflow object storage to interact with cloud storage in an ML pipeline](https://astronomer-preview.mintlify.app/docs/learn/airflow-object-storage-tutorial.md): Learn how to use Airflow object storage. - [Orchestrate OpenAI operations with Apache Airflow](https://astronomer-preview.mintlify.app/docs/learn/airflow-openai.md): Learn how to integrate OpenAI and Airflow. - [Integrate OpenLineage and Airflow](https://astronomer-preview.mintlify.app/docs/learn/airflow-openlineage.md): Learn about OpenLineage concepts and benefits of integrating with Airflow. - [Orchestrate OpenSearch operations with Apache Airflow](https://astronomer-preview.mintlify.app/docs/learn/airflow-opensearch.md): Learn how to integrate OpenSearch and Airflow. - [Create and use params in Airflow](https://astronomer-preview.mintlify.app/docs/learn/airflow-params.md): Create and use DAG and task-level params in Airflow. - [Partitioned Dag runs and asset events in Apache Airflow®](https://astronomer-preview.mintlify.app/docs/learn/airflow-partitioned-runs.md): Partition Dag runs and asset events in Airflow. - [Pass data between tasks](https://astronomer-preview.mintlify.app/docs/learn/airflow-passing-data-between-tasks.md): Learn more about the most common methods to implement data sharing between your Airflow tasks, including an in-depth explanation of XCom. - [Orchestrate pgvector operations with Apache Airflow](https://astronomer-preview.mintlify.app/docs/learn/airflow-pgvector.md): Learn how to integrate pgvector and Airflow. - [Orchestrate Pinecone operations with Apache Airflow](https://astronomer-preview.mintlify.app/docs/learn/airflow-pinecone.md): Learn how to integrate Pinecone and Airflow. - [Airflow pools](https://astronomer-preview.mintlify.app/docs/learn/airflow-pools.md): Use pools to control Airflow task parallelism. - [Orchestrate semantic querying in Qdrant with Airflow](https://astronomer-preview.mintlify.app/docs/learn/airflow-qdrant.md): Learn how to integrate Qdrant and Airflow. - [Apache Airflow® ETL Quickstart](https://astronomer-preview.mintlify.app/docs/learn/airflow-quickstart-etl.md): Apache Airflow® ETL Quickstart - [Apache Airflow® GenAI Quickstart](https://astronomer-preview.mintlify.app/docs/learn/airflow-quickstart-genai.md): Apache Airflow® GenAI Quickstart - [Orchestrate Ray jobs with Apache Airflow®](https://astronomer-preview.mintlify.app/docs/learn/airflow-ray.md): Learn how to use the Ray provider package to orchestrate Ray jobs with Apache Airflow®. - [Orchestrate Redshift operations with Airflow](https://astronomer-preview.mintlify.app/docs/learn/airflow-redshift.md): Orchestrate Redshift queries from your Airflow DAGs. - [Train a machine learning model with SageMaker and Airflow](https://astronomer-preview.mintlify.app/docs/learn/airflow-sagemaker.md): Follow a step-by-step tutorial for using Airflow to orchestrate the training and testing of a SageMaker model. - [Scaling Airflow to optimize performance](https://astronomer-preview.mintlify.app/docs/learn/airflow-scaling-workers.md): See which parameters to modify when scaling up data pipelines to make the most of Airflow. Learn about the environment, DAG, and task-level settings. - [Use setup and teardown tasks in Airflow](https://astronomer-preview.mintlify.app/docs/learn/airflow-setup-teardown.md): Learn how to use setup and teardown tasks to manage task resources in Airflow. - [Orchestrate Snowflake Queries with Airflow](https://astronomer-preview.mintlify.app/docs/learn/airflow-snowflake.md): Get enhanced observability and compute savings while orchestrating Snowflake jobs from your Airflow DAGs. - [Orchestrate Snowpark Machine Learning Workflows with Apache Airflow](https://astronomer-preview.mintlify.app/docs/learn/airflow-snowpark.md): Learn how to integrate Snowpark and Airflow. - [Using Airflow to Execute SQL](https://astronomer-preview.mintlify.app/docs/learn/airflow-sql.md): Learn the best practices for executing SQL from your DAG. Get to know Airflow’s SQL-related operators and see how to use Airflow for common SQL use cases. - [Run data quality checks using SQL check operators](https://astronomer-preview.mintlify.app/docs/learn/airflow-sql-data-quality.md): Learn how to use the SQLColumnCheckOperator, SQLTableCheckOperator and SQLCheckOperator. - [Synchronous dag execution](https://astronomer-preview.mintlify.app/docs/learn/airflow-synchronous-dag-execution.md): Learn about synchronous dag execution in Airflow. - [Airflow trigger rules](https://astronomer-preview.mintlify.app/docs/learn/airflow-trigger-rules.md): Learn about available trigger rules and how to use them. - [An introduction to the Airflow UI](https://astronomer-preview.mintlify.app/docs/learn/airflow-ui.md): Explore the Airflow UI, which helps you monitor and troubleshoot your data pipelines. Learn about some of its key features and visualizations. - [Upgrade from Apache Airflow® 2 to 3](https://astronomer-preview.mintlify.app/docs/learn/airflow-upgrade-2-3.md): Learn how upgrade your Airflow 2 environment to Airflow 3. - [Use Airflow variables](https://astronomer-preview.mintlify.app/docs/learn/airflow-variables.md): Create and use Airflow variables. - [Orchestrate Weaviate operations with Apache Airflow](https://astronomer-preview.mintlify.app/docs/learn/airflow-weaviate.md): Learn how to integrate Weaviate and Airflow. - [Manage your ML models with Weights and Biases and Airflow](https://astronomer-preview.mintlify.app/docs/learn/airflow-weights-and-biases.md): Learn how to use Airflow and Weights and Biases to manage and visualize your ML model lifecycle. - [Get started with Astro Observe](https://astronomer-preview.mintlify.app/docs/learn/astro-observe-quickstart.md): Learn how to get started with Astro Observe - [Set up Remote Execution Agents on Astro](https://astronomer-preview.mintlify.app/docs/learn/astro-remote-execution.md): Learn how to set up Remote Execution on Astro. - [Using the BashOperator](https://astronomer-preview.mintlify.app/docs/learn/bashoperator.md): Learn how to use the BashOperator to run bash commands and bash scripts. Review examples of how to run scripts in languages other than Python. - [Blueprint: A Dag writing abstraction with a no-code interface in the Astro IDE](https://astronomer-preview.mintlify.app/docs/learn/blueprint-overview.md): Learn about blueprint, a template-based system for composing Apache Airflow Dags from reusable Python templates and YAML configuration. - [How to use blueprint on Astro to write Apache Airflow® Dags in a no-code interface](https://astronomer-preview.mintlify.app/docs/learn/blueprint-user-tutorial.md): Learn how to build data pipelines from pre-built templates using the Blueprint visual builder in the Astro IDE. - [How to write blueprint templates](https://astronomer-preview.mintlify.app/docs/learn/blueprint-writer-tutorial.md): Learn how to define reusable blueprint templates in Python and compose them into Apache Airflow Dags using YAML. - [Clean up the Airflow metadata database using Dags](https://astronomer-preview.mintlify.app/docs/learn/cleanup-dag-tutorial.md): Learn how to remove unnecessary data from the Airflow metadata database by calling an Airflow Plugin from a Dag. - [Manage connections in Apache Airflow](https://astronomer-preview.mintlify.app/docs/learn/connections.md): Learn how to set up, manage, and maintain different types of connections in Apache Airflow. Use example connection configurations as the basis for your own connections. - [Create an Azure Blob Storage connection in Airflow](https://astronomer-preview.mintlify.app/docs/learn/connections/azure-blob-storage.md): Learn how to create an Azure Blob Storage connection in Airflow. - [Create a BigQuery connection in Airflow](https://astronomer-preview.mintlify.app/docs/learn/connections/bigquery.md): Learn how to create a BigQuery connection in Airflow. - [Create a Databricks connection in Airflow](https://astronomer-preview.mintlify.app/docs/learn/connections/databricks.md): Learn how to create a Databricks connection in Airflow. - [Create a dbt Cloud connection in Airflow](https://astronomer-preview.mintlify.app/docs/learn/connections/dbt-cloud.md): Learn how to create a dbt Cloud connection in Airflow. - [Create a Microsoft Entra Workload ID connection in Airflow](https://astronomer-preview.mintlify.app/docs/learn/connections/entra-workload-identity.md): Learn how to create an Azure Workload Identity connection in Airflow. - [Create a Microsoft SQL Server connection in Airflow](https://astronomer-preview.mintlify.app/docs/learn/connections/ms-sqlserver.md): Learn how to create a Microsoft SQL Server connection in Airflow. - [Create a PostgreSQL connection in Airflow](https://astronomer-preview.mintlify.app/docs/learn/connections/postgres.md): Learn how to create a PostgreSQL connection in Airflow. - [Create a Redshift connection in Airflow](https://astronomer-preview.mintlify.app/docs/learn/connections/redshift.md): Learn how to create a Redshift connection in Airflow. - [Create a Snowflake Connection in Airflow](https://astronomer-preview.mintlify.app/docs/learn/connections/snowflake.md): Learn how to create a Snowflake connection in Airflow. - [Cross-DAG dependencies](https://astronomer-preview.mintlify.app/docs/learn/cross-dag-dependencies.md): How to implement dependencies between your Airflow DAGs. - [Create DAG documentation in Apache Airflow](https://astronomer-preview.mintlify.app/docs/learn/custom-airflow-ui-docs-tutorial.md): Use Apache Airflow's built-in documentation features to generate documentation for your DAGs in the Airflow UI. - [Strategies for custom XCom backends in Airflow](https://astronomer-preview.mintlify.app/docs/learn/custom-xcom-backend-strategies.md): Use this guide to learn about different ways you can set up custom XCom backends. - [Set up a custom XCom backend using object storage](https://astronomer-preview.mintlify.app/docs/learn/custom-xcom-backends-tutorial.md): Use this tutorial to learn how to set up a custom XCom backend with object storage. - [DAG writing best practices in Apache Airflow](https://astronomer-preview.mintlify.app/docs/learn/dag-best-practices.md): Keep up to date with the best practices for developing efficient, secure, and scalable DAGs using Airflow. Learn about DAG design and data orchestration. - [Use DAG Factory to create dags](https://astronomer-preview.mintlify.app/docs/learn/dag-factory.md): Learn how to dynamically convert YAML files into Apache Airflow® dags with DAG Factory, an open source project that makes creating dags easy. - [Introduction to Apache Airflow® Dags](https://astronomer-preview.mintlify.app/docs/learn/dags.md): Learn how to write Dags and get tips on how to define an Apache Airflow® Dag in Python. Learn all about Dag parameters and their settings. - [How data products translate pipelines into business value](https://astronomer-preview.mintlify.app/docs/learn/data-products.md): Understand data products and how observability transforms invisible data work into measurable business value. - [Data quality and Airflow](https://astronomer-preview.mintlify.app/docs/learn/data-quality.md): Check the quality of your data using Airflow, Astro Observe, and third-party frameworks. - [Debug DAGs](https://astronomer-preview.mintlify.app/docs/learn/debugging-dags.md): Troubleshoot Airflow DAGs - [Running asynchronous processes in Apache Airflow®](https://astronomer-preview.mintlify.app/docs/learn/deferrable-operators.md): Run asynchronous Python in Airflow tasks and use deferrable operators to free up worker resources. - [Create dynamic Airflow tasks](https://astronomer-preview.mintlify.app/docs/learn/dynamic-tasks.md): How to dynamically create tasks at runtime in your Airflow DAGs. - [Dynamically generate DAGs in Airflow](https://astronomer-preview.mintlify.app/docs/learn/dynamically-generating-dags.md): Get to know the best ways to dynamically generate DAGs in Apache Airflow. Use examples to generate DAGs using single- and multiple-file methods. - [Manage Apache Airflow® Dag notifications](https://astronomer-preview.mintlify.app/docs/learn/error-notifications-in-airflow.md): Master the basics of Apache Airflow® notifications. Learn how to set up automatic email and Slack notifications to be alerted of events in your Dags. - [Microsoft Teams notifications](https://astronomer-preview.mintlify.app/docs/learn/example-ms-teams-callback.md): Configure notifications in Microsoft teams for DAG runs and tasks using Airflow callbacks. - [Execute a Jupyter notebook with Airflow](https://astronomer-preview.mintlify.app/docs/learn/execute-notebooks.md): Run a parameterized Jupyter notebook using Airflow and the Astro CLI. - [Intro to Airflow tutorial: Get started and run your first pipeline](https://astronomer-preview.mintlify.app/docs/learn/get-started-with-airflow.md): A hands-on Airflow tutorial to help you run your first data pipeline. Learn core concepts and build ETL workflows with Apache Airflow in minutes. - [Glossary](https://astronomer-preview.mintlify.app/docs/learn/glossary.md): A list of key Apache Airflow, observability, and other data terms and definitions that can help you learn important concepts. - [Introduction to Apache Airflow®: A Technical Overview for Beginners](https://astronomer-preview.mintlify.app/docs/learn/intro-to-airflow.md): Start learning Apache Airflow with this comprehensive guide. Covers Dags, tasks, scheduling, and all core concepts. Includes code examples and links to relevant tutorials. - [Introduction to data pipelines](https://astronomer-preview.mintlify.app/docs/learn/intro-to-data-pipelines.md): Learn what data pipelines are, how they work, and why they matter. Covers pipeline components, types, and best practices for building reliable data workflows. - [Use the KubernetesPodOperator](https://astronomer-preview.mintlify.app/docs/learn/kubepod-operator.md): Use the KubernetesPodOperator in Airflow to run tasks in Kubernetes Pods - [Airflow logging](https://astronomer-preview.mintlify.app/docs/learn/logging.md): An introduction to Airflow logging. - [Manage Airflow code](https://astronomer-preview.mintlify.app/docs/learn/managing-airflow-code.md): Learn best practices for Airflow project organization, such as when to separate out DAGs into multiple projects and how to manage code used across different projects. - [Manage task and task group dependencies in Airflow](https://astronomer-preview.mintlify.app/docs/learn/managing-dependencies.md): Learn how to manage dependencies between tasks and TaskGroups in Apache Airflow, including how to set dynamic dependencies. - [Integrate OpenLineage and Airflow with Marquez](https://astronomer-preview.mintlify.app/docs/learn/marquez.md): Use OpenLineage and Marquez to get lineage metadata locally from your Airflow DAGs. - [Learn Airflow 3](https://astronomer-preview.mintlify.app/docs/learn/overview.md): Use tutorials and concepts to learn everything you need to know about Apache Airflow® 3 and Astro - [Implementing proactive alerting](https://astronomer-preview.mintlify.app/docs/learn/proactive-alerting.md): Learn about implementing proactive alerting as part of your observability solution. - [Develop Airflow DAGs locally with PyCharm](https://astronomer-preview.mintlify.app/docs/learn/pycharm-local-dev.md): Integrate the Astro CLI with PyCharm for local development. - [AI-powered education operations with Apache Airflow®](https://astronomer-preview.mintlify.app/docs/learn/reference-architecture-ai-education-operations.md): Learn how to build an event-driven, multi-agent AI system that automates ticket processing, customer reporting, and conversational data queries using Apache Airflow. - [Context graphs for self-improving AI Agents with Apache Airflow®](https://astronomer-preview.mintlify.app/docs/learn/reference-architecture-context-graph.md): Learn how to build a context graph architecture that captures business decision traces and makes them available to AI agents as institutional memory. - [ELT with BigQuery, dbt, and Apache Airflow® for eCommerce](https://astronomer-preview.mintlify.app/docs/learn/reference-architecture-elt-bigquery-dbt.md): Learn how to build an end-to-end ELT pipeline with Apache Airflow®, BigQuery, and dbt Core. - [ELT with Snowflake and Apache Airflow® for eCommerce](https://astronomer-preview.mintlify.app/docs/learn/reference-architecture-elt-snowflake.md): Learn how to build an end-to-end ELT pipeline with Apache Airflow® and Snowflake. - [ETL with DuckDB and Apache Airflow® for travel analytics](https://astronomer-preview.mintlify.app/docs/learn/reference-architecture-etl-duckdb.md): Learn how to build an ETL pipeline with Apache Airflow®, DuckDB, and built-in data quality gates. - [Processing User Feedback: an LLM-fine-tuning reference architecture with Ray on Anyscale](https://astronomer-preview.mintlify.app/docs/learn/reference-architecture-fine-tuning-anyscale.md): Learn how to fine-tune an LLM to process and categorize user feedback with Airflow and Ray on Anyscale. - [Hybrid Search for eCommerce reference architecture](https://astronomer-preview.mintlify.app/docs/learn/reference-architecture-hybrid-search.md): Learn how to build a hybrid search application with Apache Airflow® and Weaviate. - [Integration patterns for Apache Kafka® and Apache Airflow®](https://astronomer-preview.mintlify.app/docs/learn/reference-architecture-kafka.md): Learn how to integrate Apache Kafka® with Apache Airflow® for real-time event processing using event-driven scheduling and the Airflow Kafka provider. - [Batch inference for product insights with Apache Airflow®](https://astronomer-preview.mintlify.app/docs/learn/reference-architecture-product-insights.md): Learn how to build a batch inference pipeline with Apache Airflow® and OpenAI. - [SnowPatrol: Snowflake Usage Anomaly Detection & Alerting System](https://astronomer-preview.mintlify.app/docs/learn/reference-architecture-snowpatrol.md): Learn how to build an anomaly detection system for Snowflake with Apache Airflow®. - [Rerun Airflow DAGs and tasks](https://astronomer-preview.mintlify.app/docs/learn/rerunning-dags.md): How to configure retries, catchup, backfill, and clear task instances in Airflow. - [Schedule DAGs in Apache Airflow®](https://astronomer-preview.mintlify.app/docs/learn/scheduling-in-airflow.md): Get to know Airflow scheduling concepts and different ways to schedule a DAG. - [How to share code between multiple Airflow projects](https://astronomer-preview.mintlify.app/docs/learn/sharing-code-multiple-projects.md): A description of the various ways to reuse and share code between multiple projects, with pros and cons of each solution. - [Run Soda Core checks with Airflow](https://astronomer-preview.mintlify.app/docs/learn/soda-data-quality.md): Learn how to orchestrate Soda Core data quality checks with your Airflow DAGs. - [Airflow task groups](https://astronomer-preview.mintlify.app/docs/learn/task-groups.md): Follow Astronomer’s step-by-step guide to use task groups for organizing tasks within the grid view of the Airflow user interface. - [Use Airflow templates](https://astronomer-preview.mintlify.app/docs/learn/templating.md): Learn about Jinja templating in Apache Airflow and see examples of how to pass dynamic information into task instances at runtime. - [Test Airflow DAGs](https://astronomer-preview.mintlify.app/docs/learn/testing-airflow.md): Learn about testing Airflow DAGs and gain insight into various types of tests — validation testing, unit testing, and data and pipeline integrity testing. - [ELT with Apache Airflow® and Databricks](https://astronomer-preview.mintlify.app/docs/learn/use-case-airflow-databricks.md): Use Airflow to orchestrate data loading and transformation in Databricks. - [Airflow plugins](https://astronomer-preview.mintlify.app/docs/learn/using-airflow-plugins.md): How to use Airflow plugins. - [Leverage SLAs for enhanced data quality monitoring](https://astronomer-preview.mintlify.app/docs/learn/using-slas.md): Learn about use cases and best practices for SLAs. - [Develop Airflow DAGs locally with VS Code](https://astronomer-preview.mintlify.app/docs/learn/vscode-local-dev.md): Integrate the Astro CLI with VS Code for local Airflow development. - [Airflow hooks](https://astronomer-preview.mintlify.app/docs/learn/what-is-a-hook.md): Learn about hooks and how they should be used in Apache Airflow. See an example of implementing two different hooks in a DAG. - [Airflow sensors](https://astronomer-preview.mintlify.app/docs/learn/what-is-a-sensor.md): Get an overview of Airflow sensors and learn best practices for implementing sensors in production. - [Airflow operators](https://astronomer-preview.mintlify.app/docs/learn/what-is-an-operator.md): Learn the basics of operators, which are the building blocks of Airflow DAGs. - [Astro Private Cloud: Upgrade Runtime](https://astronomer-preview.mintlify.app/docs/runtime/manage-airflow-versions.md): Adjust and upgrade Airflow versions on Astro Private Cloud. - [Astronomer Runtime overview](https://astronomer-preview.mintlify.app/docs/runtime/overview.md): Overview of Astronomer Runtime - the distribution of Apache Airflow that powers Astro. - [Astro Runtime architecture](https://astronomer-preview.mintlify.app/docs/runtime/runtime-image-architecture.md): Reference documentation for Astro Runtime, a differentiated distribution of Apache Airflow. - [Astro Runtime provider package reference](https://astronomer-preview.mintlify.app/docs/runtime/runtime-provider-reference.md): View the provider packages and versions included in each release of Astro Runtime - [Astro Runtime release notes](https://astronomer-preview.mintlify.app/docs/runtime/runtime-release-notes.md): A summary of the latest Astro Runtime features and functionality. Astro Runtime is a Docker image built by Astronomer that provides a differentiated Apache Airflow experience and execution framework. - [Astro Runtime maintenance and lifecycle policy](https://astronomer-preview.mintlify.app/docs/runtime/runtime-version-lifecycle-policy.md): Learn how Astronomer releases and maintains versions of Astro Runtime. - [Astro: Upgrade Runtime](https://astronomer-preview.mintlify.app/docs/runtime/upgrade-astro-runtime.md): Learn how to upgrade the Astro Runtime version on your Deployments. To take advantage of new features and bug and security fixes, upgrade Astro Runtime when a new version becomes available. - [Version upgrade considerations](https://astronomer-preview.mintlify.app/docs/runtime/version-upgrade-considerations.md): Reference documentation for Astro Runtime, a differentiated distribution of Apache Airflow. - [Welcome](https://astronomer-preview.mintlify.app/index.md): Learn everything you need to know about Astronomer and Apache Airflow. ## OpenAPI Specs - [astro-api-v1beta1](https://astronomer-preview.mintlify.app/openapi/astro-api-v1beta1.yaml) - [astro-api-v1](https://astronomer-preview.mintlify.app/openapi/astro-api-v1.yaml) - [openapi](https://astronomer-preview.mintlify.app/openapi.yaml) ## Optional - [Support](https://support.astronomer.io) - [Status](https://status.astronomer.io) - [Blog](https://www.astronomer.io/blog/)