Install Remote Execution Agents in a restricted kubernetes namespace

Install Remote Execution Agents in a restricted kubernetes namespace

Airflow 3

This feature is only available for Airflow 3.x Deployments.

You can install the Remote Execution Agent in a Kubernetes namespace with restricted pod security standards. Your organization might have different security standards for infrastructure supporting internal-only sandboxes compared to production environments.

Kubernetes Pod Security Standards define different security levels for Pods:

• Privileged: No restrictions (least secure)
• Baseline: Prevents known privilege escalations
• Restricted: Highly-constrained settings following security best practices (most secure)

The Restricted profile enforces the following limitations:

• Runs containers as non-root users
• Prevents privilege escalation
• Drops all Linux capabilities
• Uses read-only root filesystems when possible
• Requires a runtime default seccomp profile

However, because of these limitations, you need to complete the following additional Remote Execution Agent configuration set up.

Step 1: Create a restricted namespace

Create a Namespace in your Kubernetes manifest with the following restricted Pod security standards:

apiVersion: v1
kind: Namespace
metadata:
  name: astro-agent-restricted
  labels:
    pod-security.kubernetes.io/enforce: restricted
    pod-security.kubernetes.io/audit: restricted
    pod-security.kubernetes.io/warn: restricted

Step 2: Configure Global Security Settings

Modify your Agent’s values.yaml file to set global security context settings that apply to all Agent components’ Pods and containers:

# Global pod security context for all components
podSecurityContext:
  seccompProfile:
    type: RuntimeDefault
  runAsUser: 50000
  fsGroup: 50000
  runAsNonRoot: true

# Global container security context for all components
containerSecurityContext:
  allowPrivilegeEscalation: false
  readOnlyRootFilesystem: true
  capabilities:
    drop:
      - ALL

Step 3: Configure component-specific settings

When using the Agent in a restricted namespace, you must configure volume mounts because:

1. The container security context sets readOnlyRootFilesystem: true
2. These directories need write access during runtime
3. Using emptyDir volumes provides isolated, writable storage that meets security requirements

Worker configuration

workers:
  - name: default-worker
    # Other worker settings...

    # Required volumes for filesystem access
    volumes:
      - name: tmp
        emptyDir: {}
      - name: task-logs
        emptyDir: {}

    volumeMounts:
      - name: tmp
        mountPath: /tmp # This folder is used by the Astro Agent to maintain the socket file
      - name: task-logs
        mountPath: /usr/local/airflow/logs # Or the configured folder that holds the task logs

Dag processor configuration

dagProcessor:
  # Other Dag processor settings...

  # Required volumes for filesystem access
  volumes:
    - name: tmp
      emptyDir: {}

  volumeMounts:
    - name: tmp
      mountPath: /tmp # This folder is used by the Astro Agent to maintain the socket file

Triggerer configuration

triggerer:
  # Other triggerer settings...

  # Required volumes for filesystem access
  volumes:
    - name: tmp
      emptyDir: {}
    - name: task-logs
      emptyDir: {}

  volumeMounts:
    - name: tmp
      mountPath: /tmp # This folder is used by the Astro Agent to maintain the socket file
    - name: task-logs
      mountPath: /usr/local/airflow/logs # Or the configured folder that holds the task logs

Step 4: (Optional) Add logging sidecar configuration

loggingSidecar:
  # Other logging sidecar settings...

  securityContext:
    allowPrivilegeEscalation: false
    readOnlyRootFilesystem: true
    capabilities:
      drop:
        - ALL
  volumeMounts:
    - name: task-logs
      mountPath: /etc/airflow/logs # Or the configured folder that holds the task logs