Configure task-level Pod resources

Learn how to configure task-level Pod resources for the KubernetesPodOperator on Astro.

Astro automatically allocates resources to Pods created by the KubernetesPodOperator. Unless otherwise specified in your task-level configuration, the amount of resources your task Pod can use is defined by your default Pod resource configuration. To optimize your resource usage, Astronomer recommends specifying compute resource requests and limits for each task.

Setup

1. Define container resources
2. Define a kubernetes.client.models.V1ResourceRequirements object and provide that to the container_resources argument of the KubernetesPodOperator. For example:
3. The following code example ensures that when this dag runs, it launches a Kubernetes Pod with exactly 800m of CPU and 3Gi of memory as long as that infrastructure is available in your Deployment. After the task finishes, the Pod terminates gracefully.
4. from airflow.configuration import conf
   from airflow.providers.cncf.kubernetes.operators.pod import KubernetesPodOperator
   from kubernetes.client import models as k8s
   
   compute_resources = k8s.V1ResourceRequirements(
       limits={"cpu": "800m", "memory": "3Gi"},
       requests={"cpu": "800m", "memory": "3Gi"}
   )
   
   namespace = conf.get("kubernetes", "NAMESPACE")
   
   KubernetesPodOperator(
       namespace=namespace,
       image="<your-docker-image>",
       cmds=["<commands-for-image>"],
       arguments=["<arguments-for-image>"],
       labels={"<pod-label>": "<label-name>"},
       name="<pod-name>",
       container_resources=compute_resources,
       task_id="<task-name>",
       get_logs=True,
       in_cluster=True,
   )
   

When you add custom labels, do not remove or modify the default Airflow labels that Astro applies to KPO Pods. See Known limitations.

On Astro Hosted, Astro automatically sets resource requests equal to limits for KubernetesPodOperator task Pods. This ensures Pods receive a Kubernetes Guaranteed Quality of Service (QoS) class, which prevents resource contention and eviction. Because Astro uses the limit values as both requests and limits, your Pods are billed based on the limits you configure, even if actual usage is lower. To avoid unexpected charges, set limits close to the resources your task requires. Check your Billing and usage to view your resource use and associated charges.