Ephemeral storage configuration

Configure temporary storage for Airflow components using Kubernetes emptyDir volumes.

Ephemeral storage in Astro Private Cloud (APC) uses Kubernetes emptyDir volumes for temporary data that doesn’t need to persist across pod restarts. This guide covers configuring ephemeral storage for Airflow components including Dags, logs, and Redis.

Ephemeral storage overview

Ephemeral storage (emptyDir volumes) provides:

• Temporary storage that exists for the lifetime of a pod.
• Faster I/O if using memory-backed storage.
• No persistence across pod restarts.
• Shared access between containers in the same pod.

APC 1.x uses readOnlyRootFilesystem, which mounts the entire /usr/local/airflow directory on an emptyDir volume for every Airflow pod. Depending on Dag size, this can consume significant ephemeral storage and exceed namespace limits. Factor this into your ephemeral storage sizing.

Configurable volumes

Dags volume (gitSync)

When using git-sync for Dag deployment, Dags are stored in an emptyDir volume.

dags:
  gitSync:
    enabled: true
    emptyDirConfig:
      sizeLimit: 2Gi
      medium: Memory  # Optional: use RAM instead of disk

Parameters:

• sizeLimit: Maximum storage size (e.g., 1Gi, 2Gi).
• medium: Storage medium.
  • "" (empty): Use node’s default storage (disk).
  • "Memory": Use RAM (tmpfs) for faster access.

Logs volume

Task logs can be stored in ephemeral storage when persistence is disabled.

logs:
  persistence:
    enabled: false
  emptyDirConfig:
    sizeLimit: 10Gi
    medium: ""  # Use disk for logs (recommended)

Recommendations:

• Use larger sizeLimit for high-volume task execution.
• Avoid medium: Memory for logs unless you have aggressive cleanup.
• Enable log groomer sidecar to prevent storage exhaustion.

Redis volume

When Redis persistence is disabled, it uses ephemeral storage.

redis:
  persistence:
    enabled: false
  emptyDirConfig:
    sizeLimit: 1Gi
    medium: Memory  # RAM-backed for performance

Configuration examples

Development environment

dags:
  gitSync:
    enabled: true
    emptyDirConfig:
      sizeLimit: 1Gi

logs:
  emptyDirConfig:
    sizeLimit: 5Gi

redis:
  emptyDirConfig:
    sizeLimit: 512Mi
    medium: Memory

Production environment

dags:
  gitSync:
    enabled: true
    emptyDirConfig:
      sizeLimit: 5Gi
      medium: ""  # Use disk storage for durability

logs:
  persistence:
    enabled: true  # Use persistent storage in production
    size: 100Gi

redis:
  persistence:
    enabled: true  # Use persistent storage in production
    size: 8Gi

High-performance configuration

For latency-sensitive workloads:

dags:
  gitSync:
    enabled: true
    emptyDirConfig:
      sizeLimit: 2Gi
      medium: Memory  # Faster Dag parsing

redis:
  emptyDirConfig:
    sizeLimit: 2Gi
    medium: Memory  # Faster task queue operations

Storage medium comparison

Medium	Speed	Persistence	Memory Impact	Use Case
"" (disk)	Moderate	Pod lifetime	None	Logs, large Dags
"Memory"	Fast	Pod lifetime	Consumes RAM	Redis, small Dags

Memory-backed volumes (medium: Memory) count against container memory limits. If the volume grows too large, pods may be OOMKilled. Size memory limits accordingly or use disk-backed storage.

Sizing guidelines

Dags volume

Dag count	Recommended size
< 50	1Gi
50-200	2Gi
200-500	5Gi
500+	10Gi

Logs volume

Task Volume	Recommended Size
< 100 tasks/day	5Gi
100-1000 tasks/day	10Gi
1000-10000 tasks/day	50Gi
10000+ tasks/day	Use persistent storage

Redis volume

Worker Count	Recommended Size
1-5 workers	512Mi
5-20 workers	1Gi
20-50 workers	2Gi
50+ workers	4Gi

Monitoring storage usage

Check volume usage

kubectl exec -n <namespace> <pod-name> -- df -h

Check memory-backed volume

kubectl exec -n <namespace> <pod-name> -- mount | grep tmpfs

Troubleshooting

Pod eviction due to storage

Symptom: Pods evicted with DiskPressure or ephemeral storage exceeded.

Cause: EmptyDir volume exceeded node’s ephemeral storage limits.

Solution:

1. Increase sizeLimit in emptyDirConfig.
2. Enable log groomer with shorter retention.
3. Switch to persistent storage.

Out of memory with memory-backed volumes

Symptom: Pods OOMKilled when using medium: Memory.

Cause: Memory-backed emptyDir counts against container memory limits.

Solution:

1. Increase container memory limits.
2. Reduce sizeLimit on memory-backed volumes.
3. Switch to disk-backed storage.

Slow Dag parsing

Symptom: Dag processing takes too long.

Cause: Disk I/O latency on Dag volume.

Solution:

1. Use medium: Memory for Dag volume.
2. Ensure sufficient sizeLimit.
3. Consider SSD-backed nodes.

Best practices

• Set explicit sizeLimit to prevent unbounded storage growth.
• Use memory sparingly for performance-critical, small volumes only.
• Monitor usage and set alerts for storage utilization.
• Use persistent storage for production logs since ephemeral storage loses logs on restart.
• Size for peak usage to account for burst workloads.
• Enable log groomer to prevent log accumulation.