https://kubernetes.fail/tags/disk/Recent content in Disk on Kubernetes FailHugoenSat, 13 Apr 2024 10:00:00 +0200https://kubernetes.fail/posts/first-post/Sat, 13 Apr 2024 10:00:00 +0200https://kubernetes.fail/posts/first-post/<h2 id="what-happened">What Happened</h2> <p>Our production Kubernetes cluster experienced a complete outage when all worker nodes ran out of disk space simultaneously. This was caused by:</p> <ol> <li>Unbounded log growth from a misconfigured application</li> <li>No disk space monitoring alerts</li> <li>Default emptyDir volumes filling up node storage</li> </ol> <h2 id="the-impact">The Impact</h2> <ul> <li>All pods became pending as nodes were marked unschedulable</li> <li>4 hours of downtime during peak traffic</li> <li>Data loss for applications using emptyDir storage</li> </ul> <h2 id="lessons-learned">Lessons Learned</h2> <ol> <li><strong>Monitor disk usage</strong>: Set up alerts at 70%, 80%, and 90% capacity</li> <li><strong>Log rotation</strong>: Configure proper log rotation for all containers</li> <li><strong>Resource limits</strong>: Set storage limits on emptyDir volumes</li> <li><strong>Multi-zone deployment</strong>: Distribute workloads across availability zones</li> </ol> <h2 id="the-fix">The Fix</h2> <p>We implemented:</p>