Configure Kerberos authentication for Airflow databases

Configure Astronomer Software to use Kerberos authentication for Airflow deployment databases.

Astronomer Software supports Kerberos authentication for Airflow deployment databases, allowing you to connect to Kerberized PostgreSQL databases. This feature is available starting in Astronomer Software 0.37.7.

Overview

Kerberos is an authentication protocol that uses tickets to allow secure authentication in network environments. In enterprise environments with strict security requirements, databases are often configured to use Kerberos authentication instead of traditional username and password authentication.

With Kerberos database support in Astronomer Software, you can:

• Connect Airflow deployments to Kerberized PostgreSQL databases
• Maintain compliance with enterprise security policies that require Kerberos authentication
• Use existing Kerberos infrastructure for database authentication

How it works

Astronomer Software uses PgBouncer as a proxy between Airflow components and the Kerberized database. When you enable Kerberos for a deployment:

1. You provide labels and environment variables for PgBouncer Pods via the Houston API when creating or updating a deployment.
2. Your Kerberos credential injection mechanism (such as a mutation webhook) uses these labels to inject Kerberos credentials (keytabs or credential refresh sidecars) into the PgBouncer Pods.
3. PgBouncer authenticates to the PostgreSQL database using GSSAPI (Kerberos protocol).
4. Airflow components connect to PgBouncer using standard authentication, and PgBouncer proxies the connection to the Kerberized database.

Architecture

The following diagram shows the architecture for Kerberos database authentication in Astronomer Software 0.37.7:

Prerequisites

Before configuring Kerberos authentication, ensure you have:

• Astronomer Software 0.37.7 or later
• A Kerberized PostgreSQL database (PostgreSQL 18 has known issues)
• Kerberos infrastructure:
  • Active Directory or MIT Kerberos KDC (Key Distribution Center)
  • Network connectivity between your Kubernetes cluster and the KDC
• A mechanism to inject Kerberos credentials into PgBouncer Pods (see Kerberos credential injection)
• A Kerberos user principal created in your Active Directory
• Houston API access to create deployments

Kerberos authentication requires the Kerberos-enabled PgBouncer image quay.io/astronomer/ap-pgbouncer-krb:1.25.0-2. The generic quay.io/astronomer/ap-pgbouncer image does not support Kerberos (GSSAPI) authentication and must not be used. You configure this image in values.yaml as part of Step 1.

If you are running an airgapped installation, you must mirror quay.io/astronomer/ap-pgbouncer-krb:1.25.0-2 to your private registry in addition to the standard 0.37.7 image list — this image is not included in the automated image list. See Configure the Docker registry used for platform images for mirroring instructions.

Responsibility model

Kerberos database authentication in Astronomer Software follows a shared responsibility model:

Astronomer responsibilities

• Providing PgBouncer images with Kerberos (GSSAPI) support
• Supporting labels and environment variables on PgBouncer Pods via the Houston API
• Maintaining deployment stability during updates

Customer responsibilities

• Setting up and managing Kerberos infrastructure (Active Directory, KDC, etc.)
• Creating and managing Kerberos user principals and keytabs
• Implementing a mechanism to inject Kerberos credentials into PgBouncer Pods (such as a mutation webhook)
• Configuring appropriate labels and environment variables via the Houston API to trigger credential injection
• Creating Kerberos users in the PostgreSQL database with appropriate permissions
• Pre-creating Airflow databases for deployments
• Managing Kerberos ticket lifecycle (renewal, rotation)

Scope and limitations

The following are supported in Astronomer Software 0.37.7 with Kerberos Authentication:

• Executor: Kubernetes Executor only
• Deployment Type: Image-based deployments only
• Database: PostgreSQL with Kerberos authentication

Future releases may expand support to other executors, deployment types, and databases.

Kerberos credential injection

You are responsible for implementing a mechanism to inject Kerberos credentials into PgBouncer Pods. One common approach is using a Kubernetes mutation webhook.

Mutation webhook approach (recommended)

A mutation webhook can automatically inject Kerberos credentials when PgBouncer Pods are created. The webhook typically:

1. Watches for Pod creation requests with specific labels that you configure via the Houston API
2. Injects a sidecar container that manages Kerberos ticket renewal
3. Mounts Kerberos configuration files (krb5.conf) and keytabs as volumes
4. Configures environment variables for Kerberos authentication

When creating a deployment, you’ll specify labels in the pgbouncerConfig section that trigger your webhook to inject the necessary credentials.

Alternative approaches

Other methods for credential injection include:

• Init containers that fetch credentials from a secret management system
• Direct volume mounts of Kerberos keytabs from Kubernetes secrets
• Service mesh sidecars

Choose the approach that best fits your organization’s security requirements and infrastructure.

For implementation guidance, contact Astronomer support or your Astronomer representative.

Step 1: Enable manual connection strings

Before creating Kerberos-enabled deployments, you must enable manual connection strings in your Astronomer configuration.

1. Open your values.yaml file.

2. Add the following configuration:

deployments:
  manualConnectionStrings:
    enabled: true
global:
  airflow:
    images:
      pgbouncer:
        repository: "quay.io/astronomer/ap-pgbouncer-krb"
        tag: "1.25.0-2"

3. Push the configuration change. See Apply a platform configuration change.

Step 2: Create the Kerberos database user

You must create a Kerberos user in your PostgreSQL database with the appropriate permissions.

1. Connect to your PostgreSQL database using a superuser account.

2. Create the Kerberos user. The username must be in the format <username>@<REALM>:

CREATE USER "[e-mail]" WITH LOGIN;

3. Grant the necessary permissions:

-- For AWS RDS with Kerberos
GRANT rds_ad TO "[e-mail]";

-- Database and schema permissions
GRANT ALL PRIVILEGES ON DATABASE postgres TO "[e-mail]";
GRANT ALL PRIVILEGES ON SCHEMA public TO "[e-mail]";

-- Allow database creation
ALTER ROLE "[e-mail]" CREATEDB;

Replace astro_user with your Kerberos username and APC.ASTRONOMER.IO with your Kerberos realm.

Step 3: (Optional) Configure PgBouncer in the control plane

If you need to use a Kerberized database for the control plane (Houston’s database), you must configure PgBouncer in the astronomer namespace.

Create PgBouncer configuration

1. Create a pgbouncer.ini file with the following contents. Replace the placeholders with your actual values:

[databases]
* = host=<rds-hostname> port=5432 user=<kerberos_user>@<kerberos_realm>

[pgbouncer]
pool_mode = transaction
listen_addr = 0.0.0.0
listen_port = 6543
idle_transaction_timeout = 60
transaction_timeout = 60
autodb_idle_timeout = 60
admin_users = postgres
client_idle_timeout = 60
server_idle_timeout = 60
track_extra_parameters = search_path
stats_users = postgres

# Authentication settings
auth_type = md5
auth_file = /etc/pgbouncer/users.txt
ignore_startup_parameters = extra_float_digits

# Kerberos settings
server_gssauth_negotiate = allow
server_krb_spn = postgres/<rds-hostname>@<kerberos_realm>

max_client_conn = 200
verbose = 2
log_disconnections = 0
log_connections = 0

# Strongly recommended for RDS
server_tls_sslmode = prefer

2. Generate a password hash for the PgBouncer users.txt file:

echo -n "md5"; echo -n "<password><user>" | md5sum | awk '{print $1}'

3. Create a users.txt file with the password hashes:

"<kerberos_user>" "<output_from_above_command>"
"postgres" "<output_from_above_command>"

The password authentication in users.txt is used for Houston to connect to PgBouncer. PgBouncer then authenticates to the PostgreSQL database using Kerberos (GSSAPI).

4. Create the Kubernetes secret:

kubectl -n astronomer create secret generic astronomer-pgbouncer-config \
  --from-file=pgbouncer.ini=./pgbouncer.ini \
  --from-file=users.txt=./users.txt \
  --dry-run=client -o yaml | kubectl apply -f -

Update Astronomer configuration

1. Open your values.yaml file.

2. Add the following PgBouncer configuration:

pgbouncer:
  enabled: true
  securityContext:
    runAsGroup: 65534
    runAsNonRoot: true
    runAsUser: 65534

global:
  pgbouncer:
    enabled: true
    extraEnv: []
    extraLabels: []
    gssSupport: true
    secretName: astronomer-pgbouncer-config
    securityContext:
      runAsGroup: 65534
      runAsUser: 65534
    servicePort: "6543"
    username: postgres

3. Update the Astronomer bootstrap secret to point to the PgBouncer service:

postgres://<username>:<password>@astronomer-pgbouncer.astronomer.svc.cluster.local:6543?pgbouncer=true&connection_limit=100&pool_timeout=60&prisma_connection_limit=100

The username and password are the user and password from the users.txt file above.

The query parameters pgbouncer=true&connection_limit=100&pool_timeout=60&prisma_connection_limit=100 are critical for Prisma to work correctly with PgBouncer. Without these parameters, the control plane may experience connection issues.

4. Upgrade the Astronomer installation. You must perform the upgrade in two steps:

   Step 1: First, upgrade with the --no-hooks flag. This installs PgBouncer in the control plane without running database migration jobs:

helm upgrade astronomer astronomer/astronomer \
  --namespace astronomer \
  -f <values_file>.yaml \
  --version <version>  \
  --no-hooks

The <version> is the Astronomer Software version you want to upgrade to (e.g., 0.37.7).

Step 2: After the upgrade completes and PgBouncer is running, run the upgrade again without the --no-hooks flag. This runs the Airflow database migration jobs:

helm upgrade astronomer astronomer/astronomer \
  --namespace astronomer \
  -f <values_file>.yaml \
  --version <version>

The <version> is the Astronomer Software version you want to upgrade to (e.g., 0.37.7).

The two-step upgrade process is necessary because:

1. The first upgrade installs PgBouncer, which is required for database connectivity when using a Kerberized database.
2. The second upgrade runs database migration hooks that depend on PgBouncer being available.

Step 4: Create an Airflow deployment database

Before creating a Kerberos-enabled deployment, you must manually create the Airflow database.

Follow these steps to create the database:

1. Create a PostgreSQL client Pod for database operations:

apiVersion: v1
kind: Pod
metadata:
  labels:
    release: astronomer
    tier: astronomer
  name: postgres-debug-client
  namespace: astronomer
spec:
  containers:
  - command:
    - /bin/bash
    - -c
    - sleep infinity
    image: postgres:16
    imagePullPolicy: Always
    name: psql-client

2. Apply the Pod:

kubectl apply -f postgres-debug-client.yaml

3. Exec into the Pod and connect to the database:

kubectl -n astronomer exec -it postgres-debug-client -- bash
psql "postgres://postgres:[e-mail]:6543"

4. Create the Airflow database:

CREATE DATABASE <deployment_name>_airflow OWNER <kerberos_user>@<kerberos_realm>;

For example:

CREATE DATABASE mydeployment_airflow OWNER "[e-mail]";

Step 5: Create a Kerberos-enabled deployment

Kerberos-enabled deployments must be created using the Houston API. They cannot be created from the Astronomer UI.

Use the upsertDeployment mutation

1. Compose your mutation payload. The following example shows the required fields for a Kerberos-enabled deployment:

{
  "cloudRole": "",
  "dagDeployment": {
    "nfsLocation": "",
    "type": "image"
  },
  "dagProcessor": {
    "replicas": 1
  },
  "kerberosEnabled": true,
  "deploymentUuid": "",
  "deployRevisionDescription": "",
  "description": "",
  "dockerconfigjson": null,
  "environmentVariables": [],
  "executor": "KubernetesExecutor",
  "image": "",
  "label": "my-kerberos-deployment",
  "metadataConnection": "",
  "skipAirflowDatabaseProvisioning": true,
  "metadataConnectionJson": {
    "protocol": "postgresql",
    "user": "[e-mail]",
    "pass": "no-pass",
    "host": "<airflow-db-host>",
    "port": 5432,
    "db": "mydeployment_airflow"
  },
  "mode": "helm",
  "namespace": "",
  "properties": {
    "extra_capacity": {
      "cpu": 0,
      "memory": 0
    }
  },
  "releaseName": "mydeployment",
  "resultBackendConnection": "",
  "resultBackendConnectionJson": {
    "protocol": "postgresql",
    "user": "[e-mail]",
    "pass": "no-pass",
    "host": "<airflow-db-host>",
    "port": 5432,
    "db": "mydeployment_airflow"
  },
  "rollbackEnabled": false,
  "runtimeVersion": "11.6.0",
  "scheduler": {
    "replicas": 1
  },
  "triggerer": {},
  "webserver": {},
  "workers": {},
  "pgbouncerConfig": {
    "labels": {
      "key": "value"
    },
    "env": [
      {"name": "KERBEROS_USER", "value": "[e-mail]"},
      {"name": "KERBEROS_PASSWORD", "value": "YourKerberosPassword"}
    ],
    "extraIniResultBackend": "user=[e-mail]",
    "extraIniMetadata": "user=[e-mail]",
    "extraIni": "server_gssauth_negotiate = allow\\nserver_krb_spn = postgres/<airflow-db-host>@APC.ASTRONOMER.IO",
    "sslmode": "prefer"
  },
  "workspaceLabel": "my-workspace",
  "workspaceUuid": "<workspace-uuid>"
}

Important configuration fields

• kerberosEnabled: Must be set to true. This enables Houston to perform Kerberos-specific validation.
• skipAirflowDatabaseProvisioning: Must be set to true because you manually create the Airflow database.
• metadataConnectionJson and resultBackendConnectionJson:
  • user: Must be in the format <username>@<REALM>
  • pass: Can be any value (e.g., "no-pass") since PgBouncer uses Kerberos for database authentication
  • host: The hostname of your Kerberized PostgreSQL database
  • db: The database name you created (e.g., mydeployment_airflow)
• pgbouncerConfig:
  • labels: Custom labels for the PgBouncer Pod. Use these labels to trigger your Kerberos credential injection mechanism (e.g., "krb-inject": "enabled" or "component": "pgbouncer").
  • env: Environment variables for the PgBouncer Pod. Your credential injection mechanism can use these to configure Kerberos authentication (e.g., KERBEROS_USER, KERBEROS_PASSWORD).
  • extraIniMetadata and extraIniResultBackend: Must specify the Kerberos user.
  • extraIni: Must include Kerberos/GSS settings:
    • server_gssauth_negotiate = allow
    • server_krb_spn = postgres/<airflow-db-host>@<kerberos_realm>
  • sslmode: Set to prefer for RDS or other TLS-enabled databases

2. Execute the mutation using the Houston API.

Step 6: Verify Kerberos authentication

After creating your deployment, verify that Kerberos authentication is working correctly.

Verify deployment creation

1. List the Pods in the deployment namespace:

kubectl -n <airflow-namespace> get pods

2. Verify that all Airflow Pods are running:

kubectl -n <airflow-namespace> get pods -l component=scheduler
kubectl -n <airflow-namespace> get pods -l component=webserver
kubectl -n <airflow-namespace> get pods -l component=pgbouncer

Verify database connectivity

1. Check the PgBouncer logs for successful connections:

kubectl -n <airflow-namespace> logs <pgbouncer-pod-name>

Look for log entries showing connections using the Kerberos principal.

2. Verify that the Airflow UI loads successfully:

   a. Log in to the Astronomer UI.

   b. Navigate to your deployment.

   c. Click Airflow UI.

If the Airflow UI loads without errors, PgBouncer is successfully connecting to the database using Kerberos authentication.

Troubleshooting

PgBouncer Pod fails to start

If the PgBouncer Pod fails to start, check the following:

• Verify that your Kerberos credential injection mechanism is configured correctly.
• Check the Pod events for error messages:
  kubectl -n <airflow-namespace> describe pod <pgbouncer-pod-name>
  
• Verify that the labels in pgbouncerConfig match what your credential injection mechanism expects.

Kerberos authentication failures

If authentication fails, verify:

• The Kerberos user exists in your Active Directory and PostgreSQL database.
• The server_krb_spn in the PgBouncer configuration matches your database hostname and realm.
• Network connectivity between the Kubernetes cluster and the KDC.
• Your Kerberos credential injection mechanism is working correctly.

Airflow UI fails to load

If the Airflow UI fails to load:

• Check the webserver and scheduler logs for database connection errors.
• Verify that the database was created with the correct owner.
• Ensure the metadataConnectionJson and resultBackendConnectionJson are correctly configured.

Additional resources

• Apply a platform configuration change
• Houston API
• Houston API examples
• Workspace permissions