Skip to main content

Documentation Index

Fetch the complete documentation index at: https://astronomer-preview.mintlify.app/llms.txt

Use this file to discover all available pages before exploring further.

Policy-based setup is available only on Astro dedicated clusters. To run images from a private registry on Astro standard clusters, follow the steps inPrivate Registry.
By default, the KubernetesPodOperator expects to pull container images that are hosted publicly. If your images are hosted on the container registry native to your cloud provider, you can grant access to the images directly.

​
Prerequisites

​
Setup

1
Add Amazon ECR repository permissions
2
If your Docker image is hosted in an Amazon ECR repository, add a permissions policy to the repository to allow the KubernetesPodOperator to pull the Docker image. You don’t need to create a Kubernetes secret, or specify the Kubernetes secret in your dag. Docker images hosted in Amazon ECR repositories can only be pulled from AWS clusters.
3
  • Log in to the Amazon ECR Dashboard and then select Menu > Repositories.
  • Click the Private tab and then click the name of the repository that hosts the Docker image.
  • Click Permissions in the left menu.
  • Click Edit policy JSON.
  • Copy and paste the following policy into the Edit JSON pane: 
    {
    "Version": "2008-10-17",
    "Statement": [
        {
            "Sid": "AllowImagePullAstro",
            "Effect": "Allow",
            "Principal": {
                "AWS": "arn:aws:iam::<AstroAccountID>:role/EKS-NodeInstanceRole-<ClusterID>"
            },
            "Action": [
                "ecr:GetDownloadUrlForLayer",
                "ecr:BatchGetImage"
            ]
        }
    ]
}
    • Replace <AstroAccountID> with your Astro AWS account ID.
    • Replace <ClusterID> with your Cluster ID. To find the Cluster ID, click Organization Settings, then click Clusters in the Organization section of the Astro UI and then select the cluster. The ID is displayed at the top, along with other information about the Astro Cluster.
  • Click Save to create a new permissions policy named AllowImagePullAstro.
    • 4
    Set up the KubernetesPodOperator
    5
    The following snippet is the minimum configuration you’ll need to create a KubernetesPodOperator task on Astro:
    6
    from airflow.configuration import conf
from airflow.providers.cncf.kubernetes.operators.pod import KubernetesPodOperator

namespace = conf.get("kubernetes", "NAMESPACE")

KubernetesPodOperator(
    namespace=namespace,
    image="<your-docker-image>",
    cmds=["<commands-for-image>"],
    arguments=["<arguments-for-image>"],
    labels={"<pod-label>": "<label-name>"},
    name="<pod-name>",
    task_id="<task-name>",
    get_logs=True,
    in_cluster=True,
)
    7
    For each instantiation of the KubernetesPodOperator, you must specify the following values:
    8
  • namespace = conf.get("kubernetes", "NAMESPACE"): Every Deployment runs on its own Kubernetes namespace within a cluster. Information about this namespace can be programmatically imported as long as you set this variable.
  • image: This is the Docker image that the operator will use to run its defined task, commands, and arguments. Astro assumes that this value is an image tag that’s publicly available on Docker Hub. To pull an image from a private registry, see Pull images from a Private Registry.
  • in_cluster: If a Connection object is not passed to the KubernetesPodOperator’s kubernetes_conn_id parameter, specify in_cluster=True to run the task in the Deployment’s Astro cluster.
    • 9
    Add Amazon ECR repository URI
    10
    Replace <your-docker-image> in the instantiation of the KubernetesPodOperator with the Amazon ECR repository URI that hosts the Docker image. To locate the URI:
    11
  • In the Amazon ECR Dashboard, click Repositories in the left menu.
  • Open the Private tab and then copy the URI of the repository that hosts the Docker image.