https://kubernetes.fail/tags/kubernetes/Recent content in Kubernetes on Kubernetes FailHugoenMon, 15 Apr 2024 09:15:00 +0200https://kubernetes.fail/posts/rbac-misconfiguration/Mon, 15 Apr 2024 09:15:00 +0200https://kubernetes.fail/posts/rbac-misconfiguration/<h2 id="security-incident">Security Incident</h2> <p>An overly permissive RBAC configuration allowed an attacker to:</p> <ol> <li>Gain access through a compromised developer account</li> <li>Escalate privileges using cluster-admin binding</li> <li>Deploy cryptocurrency mining containers</li> <li>Access sensitive data from other namespaces</li> </ol> <h2 id="how-it-happened">How It Happened</h2> <ul> <li>Default namespace had cluster-admin role binding</li> <li>No network policies between namespaces</li> <li>No pod security policies enforced</li> <li>Developer accounts had excessive permissions</li> </ul> <h2 id="immediate-actions">Immediate Actions</h2> <ol> <li>Revoked all compromised credentials</li> <li>Deleted malicious pods</li> <li>Isolated affected nodes</li> <li>Rotated all cluster certificates</li> </ol> <h2 id="long-term-fixes">Long-term Fixes</h2> <ul> <li>Implemented principle of least privilege</li> <li>Enabled network policies</li> <li>Deployed pod security admission</li> <li>Regular RBAC audits</li> <li>Developer security training</li> </ul>https://kubernetes.fail/posts/network-outage/Sun, 14 Apr 2024 14:30:00 +0200https://kubernetes.fail/posts/network-outage/<h2 id="the-incident">The Incident</h2> <p>A network partition between availability zones caused our etcd cluster to split into two separate quorums, resulting in:</p> <ul> <li>Inconsistent cluster state</li> <li>Scheduling conflicts</li> <li>Data corruption in stateful applications</li> </ul> <h2 id="root-cause">Root Cause</h2> <ul> <li>Network provider had a routing issue between zones</li> <li>Our etcd cluster was spread across 3 zones but had no zone-aware configuration</li> <li>No network partition detection or automatic fencing</li> </ul> <h2 id="resolution">Resolution</h2> <ol> <li>Manually identified the healthy partition</li> <li>Forced etcd to use only the healthy members</li> <li>Rebuilt the unhealthy nodes from backups</li> <li>Implemented zone-aware etcd configuration</li> </ol> <h2 id="prevention">Prevention</h2> <ul> <li>Configure etcd with strict zone awareness</li> <li>Implement network health checks</li> <li>Regular disaster recovery drills</li> </ul>https://kubernetes.fail/posts/first-post/Sat, 13 Apr 2024 10:00:00 +0200https://kubernetes.fail/posts/first-post/<h2 id="what-happened">What Happened</h2> <p>Our production Kubernetes cluster experienced a complete outage when all worker nodes ran out of disk space simultaneously. This was caused by:</p> <ol> <li>Unbounded log growth from a misconfigured application</li> <li>No disk space monitoring alerts</li> <li>Default emptyDir volumes filling up node storage</li> </ol> <h2 id="the-impact">The Impact</h2> <ul> <li>All pods became pending as nodes were marked unschedulable</li> <li>4 hours of downtime during peak traffic</li> <li>Data loss for applications using emptyDir storage</li> </ul> <h2 id="lessons-learned">Lessons Learned</h2> <ol> <li><strong>Monitor disk usage</strong>: Set up alerts at 70%, 80%, and 90% capacity</li> <li><strong>Log rotation</strong>: Configure proper log rotation for all containers</li> <li><strong>Resource limits</strong>: Set storage limits on emptyDir volumes</li> <li><strong>Multi-zone deployment</strong>: Distribute workloads across availability zones</li> </ol> <h2 id="the-fix">The Fix</h2> <p>We implemented:</p>