First Kubernetes Failure: Node Out of Disk

Adam

- One minute read - 130 words

What Happened

Our production Kubernetes cluster experienced a complete outage when all worker nodes ran out of disk space simultaneously. This was caused by:

  1. Unbounded log growth from a misconfigured application
  2. No disk space monitoring alerts
  3. Default emptyDir volumes filling up node storage

The Impact

  • All pods became pending as nodes were marked unschedulable
  • 4 hours of downtime during peak traffic
  • Data loss for applications using emptyDir storage

Lessons Learned

  1. Monitor disk usage: Set up alerts at 70%, 80%, and 90% capacity
  2. Log rotation: Configure proper log rotation for all containers
  3. Resource limits: Set storage limits on emptyDir volumes
  4. Multi-zone deployment: Distribute workloads across availability zones

The Fix

We implemented:

  • Prometheus alerts for node disk usage
  • Fluentd with log rotation configuration
  • Resource quotas for storage
  • Regular storage capacity planning reviews