The Incident

A network partition between availability zones caused our etcd cluster to split into two separate quorums, resulting in:

• Inconsistent cluster state
• Scheduling conflicts
• Data corruption in stateful applications

Root Cause

• Network provider had a routing issue between zones
• Our etcd cluster was spread across 3 zones but had no zone-aware configuration
• No network partition detection or automatic fencing

Resolution

1. Manually identified the healthy partition
2. Forced etcd to use only the healthy members
3. Rebuilt the unhealthy nodes from backups
4. Implemented zone-aware etcd configuration

Prevention

• Configure etcd with strict zone awareness
• Implement network health checks
• Regular disaster recovery drills