Data plane clusters in Astro Private Cloud

Understand what a data plane cluster is in Astro Private Cloud, why it matters, and how to register, update, and deregister data plane clusters.

In Astro Private Cloud (APC), a cluster is a data plane Kubernetes cluster, that runs Airflow Deployments and their runtime components. You can attach multiple data plane clusters to a single control plane, which allows you to centralize management of users, workspaces, configuration, Astro UI/API, metrics, and orchestration services.

APC can run in split mode, where the data plane and control plane use separate Kubernetes clusters. Or, APC can also run in unified mode, where a single Kubernetes cluster performs both roles.

Using specific Kubernetes environments for control and data plane functions provides the following performance and administration benefits:

• Isolation and compliance: Keep teams, environments, or regions separate to meet security and governance requirements.
• Performance and scale: Allocate resources independently and scale data planes without affecting control-plane availability.
• Cost and ownership: Attribute spend to the teams or business units that own each data plane.
• Reliability boundaries: Limit the blast radius of failures and define clear service level objectives (SLO) per cluster.
• Network boundaries and residency: Keep data within specific networks or geographies.

Example

A company operates a single APC control plane and two data plane clusters at the same organization:

• Finance data plane cluster: Runs Airflow for data reconciliation and reporting with restricted network access and stricter compliance controls.
• Engineering data plane cluster: Runs Airflow for product analytics and machine learning feature pipelines with more flexible networking.

Both data planes are registered to the same control plane. Platform administrators manage users, Workspaces, and Deployments centrally, while teams operate their Dags within their isolated data planes.

Manage clusters

• Register a cluster: Register a data plane cluster.
• Override cluster configurations: Update data plane cluster details, like name, labels, connection settings, or credentials, and override cluster default Deployment configurations.
• Deregister a cluster: Deregister a data plane cluster you no longer need.
• Troubleshoot registration: Diagnose and resolve errors during data plane registration.

If you are starting fresh, set up your control plane first, then register one or more data plane clusters before creating Airflow Deployments.