Configure liveness and readiness probes

Configure liveness and readiness probes for your containers.

In Astronomer Software, you can create liveness and readiness probes to assess whether your Kubernetes Pods or network are healthy and can process requests.

Some components in Astronomer Software include liveness and readiness probes by default, but all components support adding and configuring them. Astronomer Software allows you to use the Kubernetes liveness and readiness probe definitions so you can monitor the state of your Pods.

Liveness probes can be useful in all cases. However, readiness probes might be most useful for the following scenarios:

• If you have network ports open on the container
• Containers that do not have open ports, but have multiple processes within the container
• A setup where a process in a container might never reach a healthy state because it is waiting for some state to be achieved.

Default probe behavior

You can use the following structure to define your probes in your values.yaml file. For example, you might want to adjust any default values by configuring the amount of time until a timeout. You can refer to some of the existing Default probe configurations.

You can add any definitions that are compatible with Kubernetes probes. However, because Kubernetes does not allow having more than one handler of probes you must be sure that you do not define probes to use both exec and httpGet. For consistency, the examples shown in the Default Astronomer Helm probe configurations use httpGet, but can use exec when appropriate.

Liveness probe templates

livenessProbe:
  enabled: true
  httpGet:
    path: /index.html
    port: 443

Readiness probe templates

readinessProbe:
  enabled: true
  httpGet:
    path: /index.html
    port: 444

Retrieve existing probe definitions

You can retrieve the default probe definitions from the Kubernetes manifest. The following example shows how to retrieve the definitions for Houston.

kubectl -n "${NAMESPACE}" get deployment -l component=houston -o yaml

This command produces a large amount of yaml output describing your Houston configuration. Within this output, is a section describing the livenessProbe, which looks like the following:

livenessProbe:
  failureThreshold: 10
  httpGet:
    path: /v1/healthz
    port: 8871
    scheme: HTTP
  initialDelaySeconds: 30
  periodSeconds: 10
  successThreshold: 1
  timeoutSeconds: 1

You can copy and paste this output into your values.yaml file for your Houston configuration, then adjust the values you want to customize. Then apply a platform config change.

Reference Helm values within your probes

Because values for the liveness and readiness probes are passed through the Helm template function, you can reference Helm values within the probes. Specifically, the livenessProbe and readiness values are rendered to yaml, then passed through the Helm template function, which renders any Helm template syntaxes into the produced yaml.

For example, instead of hardcoding values for your probes to match values defined by other configurations in your values.yaml file, you can use the configuration variable itself.

The following example, using the alertsmanager yaml configuration, shows how the path and ports are defined by Values.ports.http and Values.prefixURL elsewhere in the values.yaml file.

readinessProbe:
  httpGet:
    path: {{ .Values.prefixURL }}/#/status
    port: {{ .Values.ports.http }}
  initialDelaySeconds: 30
  timeoutSeconds: 30

Default Astronomer Helm probe configurations

The following components have their default probe configuration defined in the Astronomer Helm chart.

If a component does not have probes defined by default, you can see which options can have custom probe configurations.

Alert manager

alertmanager_auth-proxy:
  livenessProbe:
    httpGet:
      path: /healthz
      port: 8084
      scheme: HTTP
    initialDelaySeconds: 10
    periodSeconds: 10
  readinessProbe:
    httpGet:
      path: /healthz
      port: 8084
      scheme: HTTP
    initialDelaySeconds: 10
    periodSeconds: 10

The following can also be configured to include liveness and readiness probes:

alertmanager:
  livenessProbe: {}

Astronomer

astronomer:
  astroUI:
    livenessProbe:
      httpGet:
        path: /
        port: 8080
      initialDelaySeconds: 10
      periodSeconds: 10
    readinessProbe:
      httpGet:
        path: /
        port: 8080
      initialDelaySeconds: 10
      periodSeconds: 10
  commander:
    livenessProbe:
      httpGet:
        path: /healthz
        port: 8880
        scheme: HTTP
      initialDelaySeconds: 10
      periodSeconds: 10
      failureThreshold: 5
      successThreshold: 1
      timeoutSeconds: 5
    readinessProbe:
      httpGet:
        path: /healthz
        port: 8880
      initialDelaySeconds: 10
      periodSeconds: 10
  houston:
    livenessProbe:
      httpGet:
        path: /v1/healthz
        port: 8871
      initialDelaySeconds: 30
      periodSeconds: 10
      failureThreshold: 10
    readinessProbe:
      httpGet:
        path: /v1/healthz
        port: 8871
      initialDelaySeconds: 30
      periodSeconds: 10
      failureThreshold: 10
  registry:
    livenessProbe:
      httpGet:
        path: /
        port: 5000
      initialDelaySeconds: 10
      periodSeconds: 10
      timeoutSeconds: 5
    readinessProbe:
      httpGet:
        path: /
        port: 5000
      initialDelaySeconds: 10
      periodSeconds: 10
      timeoutSeconds: 5

The following can also be configured to include liveness and readiness probes:

astronomer:
  configSyncer:
    livenessProbe: {}
    readinessProbe: {}
  houston:
    bootstrapper:
      livenessProbe: {}
      readinessProbe: {}
    cleanupAirflowDb:
      livenessProbe: {}
      readinessProbe: {}
    cleanupDeployRevisions:
      livenessProbe: {}
      readinessProbe: {}
    cleanupDeployments:
      livenessProbe: {}
      readinessProbe: {}
    dbMigration:
      livenessProbe: {}
      readinessProbe: {}
    taskUsageMetrics:
      livenessProbe: {}
      readinessProbe: {}
    updateCheck:
      livenessProbe: {}
      readinessProbe: {}
    updateResourceStrategy:
      livenessProbe: {}
      readinessProbe: {}
    updateRuntimeCheck:
      livenessProbe: {}
      readinessProbe: {}
    upgradeDeployments:
      livenessProbe: {}
      readinessProbe: {}
    waitForDB:
      livenessProbe: {}
      readinessProbe: {}
    worker:
      livenessProbe: {}
      readinessProbe: {}

Elasticsearch

elasticsearch:
  client:
    livenessProbe:
      httpGet:
        path: /_cluster/health?local=true
        port: 9200
      initialDelaySeconds: 90
    readinessProbe:
      httpGet:
        path: /_cluster/health?local=true
        port: 9200
      initialDelaySeconds: 5
  exporter:
    livenessProbe:
      httpGet:
        path: /healthz
        port: http
      initialDelaySeconds: 30
      timeoutSeconds: 10
    readinessProbe:
      httpGet:
        path: /healthz
        port: http
      initialDelaySeconds: 10
      timeoutSeconds: 10
  master:
    livenessProbe:
      tcpSocket:
        port: 9300
    readinessProbe:
      httpGet:
        path: /_cluster/health?local=true
        port: 9200
      initialDelaySeconds: 5

The following components do not have probes configured by default:

elasticsearch:
  curator:
    livenessProbe:
      exec:
        command:
        - /bin/true
    readinessProbe:
      exec:
        command:
        - /bin/true
  data:
    readinessProbe:
      exec:
        command:
        - /bin/true
  nginx:
    livenessProbe:
      exec:
        command:
        - /bin/true
    readinessProbe:
      exec:
        command:
        - /bin/true
  sysctlInitContainer:
    livenessProbe:
      exec:
        command:
        - /bin/true
    readinessProbe:
      exec:
        command:
        - /bin/true

External-es-proxy

The following components do not have probes configured by default:

external-es-proxy:
  awsproxy:
    livenessProbe:
      exec:
        command:
        - /bin/true
    readinessProbe:
      exec:
        command:
        - /bin/true
  livenessProbe:
    exec:
      command:
      - /bin/true
  readinessProbe:
    exec:
      command:
      - /bin/true

Fluentd

livenessProbe:
  exec:
    command:
      - /bin/bash
      - -c
      - >-
        if (( $(ruby -e "require 'net/http';require 'uri';uri = URI.parse('http://127.0.0.1:24231/metrics');response = Net::HTTP.get_response(uri);puts response.body" | grep 'fluentd_output_status_buffer_queue_length{' | awk '{ print ($NF > 8) }') )); then exit 1; fi; exit 0
  failureThreshold: 3
  initialDelaySeconds: 30
  periodSeconds: 15
  successThreshold: 1
  timeoutSeconds: 5

The following components do not have probes configured by default:

fluentd:
  readinessProbe:
    exec:
      command:
      - /bin/true

Global

The following components do not have probes configured by default:

global:
  loggingSidecar:
      readinessProbe: {}
      livenessProbe: {}
  dagOnlyDeployment:
      server:
          readinessProbe: {}
          livenessProbe: {}
      client:
          readinessProbe: {}
          livenessProbe: {}
    authSidecar:
      readinessProbe: {}
      livenessProbe: {}

Grafana

grafana:
  livenessProbe:
    httpGet:
      path: /api/health
      port: 3000
    initialDelaySeconds: 10
    periodSeconds: 10
  readinessProbe:
    httpGet:
      path: /api/health
      port: 3000
    initialDelaySeconds: 10
    periodSeconds: 10

The following components do not have probes configured by default:

grafana:
  bootstrapper:
    livenessProbe: {}
    readinessProbe: {}
  waitForDB:
    livenessProbe: {}
    readinessProbe: {}

Kibana

kibana:
  livenessProbe:
    httpGet:
      path: /healthz
      port: 8084
      scheme: HTTP
    initialDelaySeconds: 10
    periodSeconds: 10
  readinessProbe:
    httpGet:
      path: /healthz
      port: 8084
      scheme: HTTP
    initialDelaySeconds: 10
    periodSeconds: 10

The following components do not have probes configured by default:

kibana:
  defaultIndexJob:
    livenessProbe:
      exec:
        command:
        - /bin/true
    readinessProbe:
      exec:
        command:
        - /bin/true
  livenessProbe:
    exec:
      command:
      - /bin/true
  readinessProbe:
    exec:
      command:
      - /bin/true

Kube-state

The following components do not have probes configured by default:

kube-state:
  readinessProbe: {}

NATS

nats:
  nats:
    livenessProbe:
      httpGet:
        path: /
        port: 8222
      initialDelaySeconds: 10
      timeoutSeconds: 5
    readinessProbe:
      httpGet:
        path: /
        port: 8222
      initialDelaySeconds: 10
      timeoutSeconds: 5

The following components do not have probes configured by default:

nats:
  exporter:
    enabled: true
    livenessProbe:
      exec:
        command:
        - /bin/true
    readinessProbe:
      exec:
        command:
        - /bin/true
  reloader:
    livenessProbe:
      exec:
        command:
        - /bin/true
    readinessProbe:
      exec:
        command:
        - /bin/true

nginx

The following components do not have probes configured by default:

  defaultBackend:
    readinessProbe:
      exec:
        command:
        - /bin/true
  readinessProbe:
    exec:
      command:
      - /bin/true

PgBouncer

pgbouncer:
  livenessProbe:
    tcpSocket:
      port: 5432
  readinessProbe:
    tcpSocket:
      port: 5432

PostgreSQL

  livenessProbe:
    exec:
      command:
        - sh
        - -c
        - exec pg_isready -U "postgres" -h 127.0.0.1 -p 5432
    initialDelaySeconds: 30
    periodSeconds: 10
    timeoutSeconds: 5
    successThreshold: 1
    failureThreshold: 6
  readinessProbe:
    exec:
      command:
        - sh
        - -c
        - -e
        - 'pg_isready -U "postgres" -h 127.0.0.1 -p 5432\n'
    initialDelaySeconds: 5
    periodSeconds: 10
    timeoutSeconds: 5
    successThreshold: 1
    failureThreshold: 6

The following components do not have probes configured by default:

postgresql:
  metrics:
    livenessProbe: {}
    readinessProbe: {}

Prometheus

prometheus:
  livenessProbe:
    httpGet:
      path: /-/healthy
      port: 9090
    initialDelaySeconds: 10
    periodSeconds: 5
    failureThreshold: 3
    timeoutSeconds: 1
  readinessProbe:
    httpGet:
      path: /-/ready
      port: 9090
    initialDelaySeconds: 10
    periodSeconds: 5
    failureThreshold: 3
    timeoutSeconds: 1
  authproxy:
      livenessProbe:
        httpGet:
          path: /healthz
          port: 8084
          scheme: HTTP
        initialDelaySeconds: 10
        periodSeconds: 10
    readinessProbe:
      httpGet:
        path: /healthz
        port: 8084
        scheme: HTTP
      initialDelaySeconds: 10
      periodSeconds: 10
prometheus-blackbox-exporter:
  livenessProbe:
    httpGet:
      path: /health
      port: http
  readinessProbe:
    httpGet:
      path: /health
      port: http
prometheus-node-exporter:
  livenessProbe:
    httpGet:
      path: /
      port: 9100
  readinessProbe:
    httpGet:
      path: /
      port: 9100
prometheus-postgres-exporter:
  livenessProbe:
    tcpSocket:
      port: 9187
    initialDelaySeconds: 5
    periodSeconds: 10
  readinessProbe:
    tcpSocket:
      port: 9187
    initialDelaySeconds: 5
    periodSeconds: 10

The following components do not have probes configured by default:

prometheus:
  configMapReloader:
    livenessProbe: {}
    readinessProbe: {}
  filesdReloader:
    livenessProbe: {}
    readinessProbe: {}

STAN

  livenessProbe:
    httpGet:
      path: /streaming/serverz
      port: monitor
    initialDelaySeconds: 10
    timeoutSeconds: 5
  readinessProbe:
    httpGet:
      path: /streaming/serverz
      port: monitor
    initialDelaySeconds: 10
    timeoutSeconds: 5

The following components do not have probes configured by default:

stan:
  exporter:
    livenessProbe:
      exec:
        command:
        - /bin/true
    readinessProbe:
      exec:
        command:
        - /bin/true
  waitForNatsServer:
    livenessProbe:
      exec:
        command:
        - /bin/true
    readinessProbe:
      exec:
        command:
        - /bin/true

Default Airflow chart probe configurations

You can also define liveness and readiness probes using the Astronomer Airflow chart.

Airflow

This includes:

• dagProcessor
• flower
• pgbouncer
• postgresql
• scheduler
• triggerer
• webserver
• workers

airflow:
  dagProcessor:
    livenessProbe:
      command: null
      failureThreshold: 5
      initialDelaySeconds: 10
      periodSeconds: 60
      timeoutSeconds: 20
    readinessProbe:
      initialDelaySeconds: 10
      timeoutSeconds: 20
      failureThreshold: 5
      periodSeconds: 60
    logGroomerSidecar:
      enabled: true
      livenessProbe:
        initialDelaySeconds: 60
        timeoutSeconds: 20
        failureThreshold: 5
        periodSeconds: 60
      readinessProbe:
        initialDelaySeconds: 60
        timeoutSeconds: 20
        failureThreshold: 5
        periodSeconds: 60
  flower:
    livenessProbe:
      failureThreshold: 10
      initialDelaySeconds: 10
      periodSeconds: 5
      timeoutSeconds: 5
    readinessProbe:
      failureThreshold: 10
      initialDelaySeconds: 10
      periodSeconds: 5
      timeoutSeconds: 5
  pgbouncer:
    metricsExporterSidecar:
      livenessProbe:
        initialDelaySeconds: 10
        periodSeconds: 10
        timeoutSeconds: 1
      readinessProbe:
        initialDelaySeconds: 10
        periodSeconds: 10
        timeoutSeconds: 1
  postgresql:
    metrics:
      customLivenessProbe: {}
      customReadinessProbe: {}
      livenessProbe:
        enabled: true
        failureThreshold: 6
        initialDelaySeconds: 5
        periodSeconds: 10
        successThreshold: 1
        timeoutSeconds: 5
      readinessProbe:
        enabled: true
        failureThreshold: 6
        initialDelaySeconds: 5
        periodSeconds: 10
        successThreshold: 1
        timeoutSeconds: 5
    primary:
      customLivenessProbe: {}
      customReadinessProbe: {}
      livenessProbe:
        enabled: true
        failureThreshold: 6
        initialDelaySeconds: 30
        periodSeconds: 10
        successThreshold: 1
        timeoutSeconds: 5
      readinessProbe:
        enabled: true
        failureThreshold: 6
        initialDelaySeconds: 5
        periodSeconds: 10
        successThreshold: 1
        timeoutSeconds: 5
    readReplicas:
      customLivenessProbe: {}
      customReadinessProbe: {}
      livenessProbe:
        enabled: true
        failureThreshold: 6
        initialDelaySeconds: 30
        periodSeconds: 10
        successThreshold: 1
        timeoutSeconds: 5
      readinessProbe:
        enabled: true
        failureThreshold: 6
        initialDelaySeconds: 5
        periodSeconds: 10
        successThreshold: 1
        timeoutSeconds: 5
  scheduler:
    livenessProbe:
      command: null
      failureThreshold: 5
      initialDelaySeconds: 10
      periodSeconds: 60
      timeoutSeconds: 30
  triggerer:
    livenessProbe:
      command: null
      failureThreshold: 5
      initialDelaySeconds: 10
      periodSeconds: 60
      timeoutSeconds: 20
  webserver:
    livenessProbe:
      failureThreshold: 5
      initialDelaySeconds: 15
      periodSeconds: 10
      scheme: HTTP
      timeoutSeconds: 5
    readinessProbe:
      failureThreshold: 5
      initialDelaySeconds: 15
      periodSeconds: 10
      scheme: HTTP
      timeoutSeconds: 5
  workers:
    livenessProbe:
      command: null
      enabled: true
      failureThreshold: 5
      initialDelaySeconds: 10
      periodSeconds: 60
      timeoutSeconds: 20

Auth sidecar

The following can also be configured to include liveness and readiness probes:

authSidecar:
  livenessProbe: {}
  readinessProbe: {}

DAG deploy server

The following can also be configured to include liveness and readiness probes:

dagDeploy:
  livenessProbe: {}
  readinessProbe: {}

Logging sidecar

The following can also be configured to include liveness and readiness probes:

loggingSidecar:
  livenessProbe: {}
  readinessProbe: {}