https://kubernetes.fail/tags/network/Recent content in Network on Kubernetes FailHugoenSun, 14 Apr 2024 14:30:00 +0200https://kubernetes.fail/posts/network-outage/Sun, 14 Apr 2024 14:30:00 +0200https://kubernetes.fail/posts/network-outage/<h2 id="the-incident">The Incident</h2> <p>A network partition between availability zones caused our etcd cluster to split into two separate quorums, resulting in:</p> <ul> <li>Inconsistent cluster state</li> <li>Scheduling conflicts</li> <li>Data corruption in stateful applications</li> </ul> <h2 id="root-cause">Root Cause</h2> <ul> <li>Network provider had a routing issue between zones</li> <li>Our etcd cluster was spread across 3 zones but had no zone-aware configuration</li> <li>No network partition detection or automatic fencing</li> </ul> <h2 id="resolution">Resolution</h2> <ol> <li>Manually identified the healthy partition</li> <li>Forced etcd to use only the healthy members</li> <li>Rebuilt the unhealthy nodes from backups</li> <li>Implemented zone-aware etcd configuration</li> </ol> <h2 id="prevention">Prevention</h2> <ul> <li>Configure etcd with strict zone awareness</li> <li>Implement network health checks</li> <li>Regular disaster recovery drills</li> </ul>