Documentation Index
Fetch the complete documentation index at: https://astronomer-preview.mintlify.app/llms.txt
Use this file to discover all available pages before exploring further.
Astro Private Cloud allows you to customize the minimum and maximum sizes of most Astronomer platform and Airflow components.
You can configure the CPU and memory resources of Airflow components through the Astro UI or with the Houston API.
You can use astronomer.houston.config.deployments.maxPodCapacity to configure the maximum size any individual pod can be.
astronomer:
houston:
config:
deployments:
maxPodCapacity:
cpu: 3500
memory: 13440
LimitRange and ResourceQuota for the namespace associated with each Deployment.
These values are automatically adjusted to account for the resource requirements of various components.
You can add additional resources, beyond the standard amount allocated based on the resource-requirements of standing components, to the LimitRange and ResourceQuota. Add resources by configuring astronomer.houston.config.deployments.maxExtraCapacity to account for the requirements of KubernetesExecutor and KubernetesPodOperator tasks.
astronomer:
houston:
config:
deployments:
maxExtraCapacity:
cpu: 40000
memory: 153600
astronomer.houston.config.deployments.components.
A list of configurable components and options is provided in Configurable Components.
KubernetesExecutor Task pod sizes are created on an as-needed basis, and don’t have persisting resource requirements. Their resource requirements areconfigured at the task level. astronomer.houston.config.deployments.components in your values.yaml with a higher limit:
When increasing CPU or memory limits, ensure themaximum pod sizeis large enough to avoid errors during pod creation. astronomer:
houston:
config:
deployments:
components:
- name: workers
resources:
default:
cpu: 1000
memory: 3840
minimum:
cpu: 100
memory: 384
limit:
cpu: 3000
memory: 11520
KubernetesExecutor:
default:
cpu: 100
memory: 384
minimum:
cpu: 100
memory: 384
limit:
cpu: 3000
memory: 11520
extra:
- name: terminationGracePeriodSeconds
default: 600
minimum: 0
limit: 36000
- name: replicas
default: 1
minimum: 1
limit: 10
# any additional component configurations go here
# - name: another-component
# resources:
# default: 10
# ...
Configurable Components
When defining components, you must include the full definition of the component in the list entry after the components key, instead of only the components you want to define.
KubernetesExecutor task pod sizes are created on an as-needed basis and don’t have persisting resource requirements. Their resource requirements areconfigured at the task level. Airflow Scheduler
- name: scheduler
resources:
default:
cpu: 500
memory: 1920
minimum:
cpu: 500
memory: 1920
limit:
cpu: 3000
memory: 11520
extra:
- name: replicas
default: 1
minimum: 1
limit: 4
Airflow Webserver
- name: webserver
resources:
default:
cpu: 500
memory: 1920
minimum:
cpu: 500
memory: 1920
limit:
cpu: 3000
memory: 11520
Airflow Apiserver (Airflow 3.0 and above)
- name: apiServer
resources:
default:
cpu: 1000
memory: 3840
minimum:
cpu: 1000
memory: 3840
limit:
cpu: 3000
memory: 11520
extra:
- name: replicas
default: 1
minimum: 1
limit: 4
#### StatsD
```yaml
- name: statsd
resources:
default:
cpu: 200
memory: 768
minimum:
cpu: 200
memory: 768
limit:
cpu: 3000
memory: 11520
Database Connection Pooler (PgBouncer)
- name: pgbouncer
resources:
default:
cpu: 200
memory: 768
minimum:
cpu: 200
memory: 768
limit:
cpu: 200
memory: 768
Celery Diagnostic Web Interface (Flower)
- name: flower
resources:
default:
cpu: 200
memory: 768
minimum:
cpu: 200
memory: 768
limit:
cpu: 200
memory: 768
Redis
- name: redis
resources:
default:
cpu: 200
memory: 768
minimum:
cpu: 200
memory: 768
limit:
cpu: 200
memory: 768
Celery Workers
- name: workers
resources:
default:
cpu: 1000
memory: 3840
minimum:
cpu: 100
memory: 384
limit:
cpu: 3000
memory: 11520
extra:
- name: terminationGracePeriodSeconds
default: 600
minimum: 0
limit: 36000
- name: replicas
default: 1
minimum: 1
limit: 10
Triggerer
- name: triggerer
resources:
default:
cpu: 500
memory: 1920
minimum:
cpu: 500
memory: 1920
limit:
cpu: 3000
memory: 11520
extra:
- name: replicas
default: 1
minimum: 0
limit: 2
#### Dag Processor
```yaml
- name: dagProcessor
resources:
default:
cpu: 500
memory: 1920
minimum:
cpu: 500
memory: 1920
limit:
cpu: 3000
memory: 11520
extra:
- name: replicas
default: 0
minimum: 0
limit: 3
## Manage Kubernetes worker CPU and memory with global platform config [#disable-kubernetes-resources-ui-api]
You can use `workers.resources.enabled` for the `KubernetesExecutor` to manage worker Pod CPU and memory allocations with a platform configuration. When set to `false`, Astro Private Cloud ignores worker Pod CPU or memory allocations set in the API or UI. This allows you to control Pod resources with your own Kubernetes policies or admission controllers.
### API/UI manages Pod resources (Default KubernetesExecutor configuration)
To configure the default behavior where the API and UI manage worker Pod CPU and memory resources for the KubernetesExecutor, use the following configuration:
```yaml
deployments:
executors:
- name: KubernetesExecutor
enabled: true
components:
- scheduler
- webserver
- apiServer
- statsd
- pgbouncer
- triggerer
- dagProcessor
defaultExtraCapacity:
cpu: 1000
memory: 3840
workers:
ephemeralStorage:
disabled: true
resources:
enabled: true
Disable API/UI resource configuration
If you configure your KubernetesExecutor so that CPU and memory requests and limits are set outside of Astro Private Cloud, you must disable resource configuration set with the Astro Private Cloud API/UI. To disable the Astro Private Cloud UI/API configuration, add the resources.enabled: false flag to your values.yaml file. This ensures that Astro Private Cloud applies the worker resource settings exclusively from your preferred source, such as Pod mutation hooks or Pod configuration files. The following configuration sets them as an empty dict.
deployments:
executors:
- name: KubernetesExecutor
enabled: true
components:
- scheduler
- webserver
- apiServer
- statsd
- pgbouncer
- triggerer
- dagProcessor
defaultExtraCapacity:
cpu: 1000
memory: 3840
workers:
ephemeralStorage:
disabled: true
resources:
enabled: false
executor_config into the Operator. When not defined, these tasks default to using 0.1 Vcpu/384 Mi of memory. This means that when you define resource requests or limits for CPU and memory, ensure the maximum pod size is large enough to avoid errors during pod creation.
Astro Private Cloud does not automatically raise the namespace-level cumulative resource limits for pods created by the KubernetesExecutor. To avoid pod creation failures, increase themaxExtraCapacityto support your desired level of resourcing and concurrency.
# import kubernetes.client.models as k8s
from kubernetes.client import models as k8s
# define an executor_config with the desired resources
my_executor_config={
"pod_override": k8s.V1Pod(
spec=k8s.V1PodSpec(
containers=[
k8s.V1Container(
name="base",
resources=k8s.V1ResourceRequirements(
requests={
"cpu": "50m",
"memory": "384Mi"
},
limits={
"cpu": "1000m",
"memory": "1024Mi"
}
)
)
]
)
)
}
# pass in executor_config=my_executor_config to any Operator
#@task(executor_config=my_executor_config)
#def some_task():
# ...
#task = PythonOperator(
# task_id="another_task",
# python_callable=my_fun,
# executor_config=my_executor_config
#)
LimitRanges and quotas defined within the pod namespace.
