Skip to main content

Documentation Index

Fetch the complete documentation index at: https://astronomer-preview.mintlify.app/llms.txt

Use this file to discover all available pages before exploring further.

You can display task logs in the Airflow UI by exporting logs to object storage and configuring the Astro API Server to retrieve them. Start by enabling log display after task completion, then optionally extend the setup to stream logs in real time as tasks run. This guide explains configuring post-task log display and expanding that configuration to support real-time log streaming.

​
Displaying task logs after task completion

Set up log uploading so logs are visible in the Airflow UI after task completion. This requires:
  • Remote Execution Agent configuration (values.yaml)
  • Astro UI Deployment configuration
  • Workload identities: write access for the Remote Execution Agent, read access for the Astro API Server
The Astro Orchestration Plane provides secure private connectivity with a pre-configured S3 Gateway Endpoint.
  1. Configure the following environment variables in the Helm chart’s values.yaml, and replace the path for the AIRFLOW__LOGGING__REMOTE_BASE_LOG_FOLDER value with your information:
commonEnv:
  - name: AIRFLOW__LOGGING__REMOTE_LOGGING
    value: "True"
  - name: AIRFLOW__LOGGING__REMOTE_LOG_CONN_ID
    value: "astro_aws_logging"
  - name: AIRFLOW_CONN_ASTRO_AWS_LOGGING
    value: "s3://"
  - name: AIRFLOW__LOGGING__REMOTE_BASE_LOG_FOLDER
    value: "s3://<bucket>/<deployment-id>"
  - name: AIRFLOW__LOGGING__LOGGING_CONFIG_CLASS
    value: "astronomer.runtime.logging.logging_config"
  - name: ASTRONOMER_ENVIRONMENT
    value: "cloud"
Mounting credentials manuallyIf you do not use workload identity and instead want to manually mount a credential, you must also add the following environment variable defining the location of a token file to your Remote Agent’s values.yaml file. You can customize the file path, /tmp/logging-token, to the name of your logging token file.
  - name: ASTRO_LOGGING_AWS_WEB_IDENTITY_TOKEN_FILE
    value: "/tmp/logging-token"
  1. Run helm upgrade to apply the change to your Agents.
  2. In the Astro UI, navigate to your Deployment and click the Details tab. Click Edit in the Advanced section to access your logging configurations.
  3. Select Bucket Storage in the Task Logs field and fill in the Bucket URL as s3://<bucket>/<deployment-id>. Or, use the path that you configured for AIRFLOW__LOGGING__REMOTE_BASE_LOG_FOLDER in your Remote Agent’s Helm chart’s values.yaml.
  4. In the Workload Identity for Bucket Storage section, select Customer Managed Identity and follow the instructions to set up your Customer Managed Identity so that the identity you create has read access to the specified bucket and path. The Customer Managed Identity must have s3:GetObject and s3:ListBucket permissions on the S3 bucket. Additionally, ensure that no ACLs on the bucket restrict those actions.
Default Identity is not currently supported for Task Logs Bucket Storage on AWS. You must use Customer Managed Identity.
  1. (Optional) If your log bucket is in a different region from your Astro Deployment, you need to define the AWS region in the AIRFLOW__ASTRONOMER_PROVIDERS_LOGGING__AWS_REGION environment variable for Astronomer-managed components. In the Astro UI, navigate to your Deployment and click the Environment tab. Click Environment Variables, then click (+) Environment Variable to add the following environment variables to your Deployment:
  • AIRFLOW__ASTRONOMER_PROVIDERS_LOGGING__AWS_REGION <The region in which the S3 bucket is configured>

​
Displaying task logs during task execution

Once you have post-completion log visibility, you can enable real-time log display. Remote Execution prevents the Airflow API server from reading logs directly from workers until they reach object storage. Use Vector, included in the Remote Execution Agent Helm chart, to upload partial logs while tasks are running.

​
Prerequisites

Before you configure Vector, ensure that your Remote Execution Deployment is already set up to upload task logs to object storage after task completion.

​
Enable Vector sidecar

Use Vector to watch for log file changes and upload updates to object storage during task execution. In your Helm values.yaml:
  • Set loggingSidecar.enabled to true:
loggingSidecar:
  enabled: true
  • Configure loggingSidecar.volumeMounts:
loggingSidecar:
  volumeMounts:
    - name: task-logs
      mountPath: /var/log/airflow/task_logs
      readOnly: true
    - name: vector-data
      mountPath: /var/lib/vector

​
Configure log upload for your cloud provider

Configure loggingSidecar.config:
loggingSidecar:
  config: |
    # Vector configuration for Astronomer Remote Execution agents for uploading Airflow task logs to AWS S3

    sources:
      airflow_task_logs:
        type: file
        include:
          - /var/log/airflow/task_logs/**/*.log
        read_from: beginning

    transforms:
      strip_path_prefix:
        type: remap
        inputs: [airflow_task_logs]
        source: |
          .log_path, err = replace(.file, "/var/log/airflow/task_logs/", "")
          if err != null {
            abort
          }

    sinks:
      s3:
        # For more Vector AWS S3 configuration options, see https://vector.dev/docs/reference/configuration/sinks/aws_s3
        type: aws_s3
        inputs: [strip_path_prefix]
        bucket: # set bucket name e.g. airflow_logs
        region: # set AWS region e.g. us-east-2
        compression: none
        encoding:
          codec: text
        key_prefix: '{{ "{{" }} log_path {{ "}}" }}.'
        filename_time_format: "%Y-%m-%dT%H-%M-%S"
        filename_append_uuid: false
        batch:
          max_bytes: 1000000  # configure based on your storage costs and log frequency requirements - see Caveats section below
          timeout_secs: 10  # configure based on your storage costs and log frequency requirements - see Caveats section below
AWS authentication with VectorAbove Vector config assumes a managed identity is set up for authentication, as described in Displaying task logs after task completion.If you require a different way to authenticate with AWS, such as static keys, see https://vector.dev/docs/reference/configuration/sinks/aws_s3/#auth for all available options.
Developing Vector Remap Language (VRL)Vector expressions are written in Vector Remap Language (VRL). If you want to edit an expression in the Vector config, this online VRL playground is a useful debugging tool.
Debugging VectorIf you’re having issues uploading logs, you can enable debug logging for the Vector sidecar by adding this to the sink configuration (so you’ll have 2 sinks, e.g. an s3 sink, and a debug sink):
debug:
  type: console
  inputs: [strip_path_prefix]
  encoding:
    codec: json
With this second sink, Vector will display debug logs on the console, accessible with kubectl logs [worker pod] -c vector-logging-sidecar.
  • Configure workers[*].volumes:
volumes:
  - name: task-logs
    emptyDir: {}
  - name: vector-data
    emptyDir: {}
  • Configure workers[*].volumeMounts:
volumeMounts:
  - name: task-logs
    mountPath: /usr/local/airflow/logs
  • Configure triggerer.volumes:
volumes:
  - name: task-logs
    emptyDir: {}
  - name: vector-data
    emptyDir: {}
  • Configure triggerer.volumeMounts:
volumeMounts:
  - name: task-logs
    mountPath: /usr/local/airflow/logs
  • Set AIRFLOW__LOGGING__DELETE_LOCAL_LOGS in commonEnv:
commonEnv:
  - name: AIRFLOW__LOGGING__DELETE_LOCAL_LOGS
    value: "True"

​
Log upload process

Partial logs are uploaded and displayed as follows:
  1. Airflow worker or triggerer writes local task log files, as set by AIRFLOW__LOGGING__LOG_FILENAME_TEMPLATE.
  2. Vector watches /var/log/airflow/task_logs/**/*.log and uploads log changes in chunks while the task runs.
  3. Vector appends a timestamp to the file name before uploading each chunk.
  4. Airflow scans object storage for log chunks when displaying the UI log view.
  5. The UI displays all log content to the user.
Version compatibilityUsing Vector to upload logs assumes Airflow’s logging format is compatible. Significant changes to Airflow logging may require reconfiguration.

​
Caveats

​
Duplicate log storage

After task completion, Airflow uploads the complete log to object storage and deletes the local copy. This causes duplication:
  1. Partial logs from Vector
  2. Complete log from Airflow
The Airflow API server deduplicates log lines by timestamp and message. Only storage usage is affected; logs are displayed once.

​
Small file problem

High-frequency, small log file uploads can create many small objects. This may increase storage costs, load on object storage, or trigger rate limits. Adjust file size and upload frequency in your Vector config to balance performance and cost.
  • AWS bills object retrieval at a 128KB minimum on certain storage classes (source).
  • A large number of small objects means more object requests (PUTs, GETs, LISTs) and more load on metadata/indexing; this can result in rate limits or latency issues.
Ensure a proper balance between filesize/timeout and log upload frequency in your Vector config.