Kubernetes Fail

Documenting real-world Kubernetes failures and the lessons we’ve learned from them.

About This Site

This site collects post-mortems, incident reports, and lessons learned from running Kubernetes in production. Our goal is to help others avoid the same mistakes we’ve made.

Recent Failures

Check out our latest posts about Kubernetes failures and lessons learned.

Categories

Contribute

Have a Kubernetes failure story to share? Submit a pull request or open an issue!

RBAC Misconfiguration Leads to Security Breach

Security Incident

An overly permissive RBAC configuration allowed an attacker to:

  1. Gain access through a compromised developer account
  2. Escalate privileges using cluster-admin binding
  3. Deploy cryptocurrency mining containers
  4. Access sensitive data from other namespaces

How It Happened

  • Default namespace had cluster-admin role binding
  • No network policies between namespaces
  • No pod security policies enforced
  • Developer accounts had excessive permissions

Immediate Actions

  1. Revoked all compromised credentials
  2. Deleted malicious pods
  3. Isolated affected nodes
  4. Rotated all cluster certificates

Long-term Fixes

  • Implemented principle of least privilege
  • Enabled network policies
  • Deployed pod security admission
  • Regular RBAC audits
  • Developer security training

By Adam

read more

Network Partition Causes Split-Brain

The Incident

A network partition between availability zones caused our etcd cluster to split into two separate quorums, resulting in:

  • Inconsistent cluster state
  • Scheduling conflicts
  • Data corruption in stateful applications

Root Cause

  • Network provider had a routing issue between zones
  • Our etcd cluster was spread across 3 zones but had no zone-aware configuration
  • No network partition detection or automatic fencing

Resolution

  1. Manually identified the healthy partition
  2. Forced etcd to use only the healthy members
  3. Rebuilt the unhealthy nodes from backups
  4. Implemented zone-aware etcd configuration

Prevention

  • Configure etcd with strict zone awareness
  • Implement network health checks
  • Regular disaster recovery drills

By Adam

read more

First Kubernetes Failure: Node Out of Disk

What Happened

Our production Kubernetes cluster experienced a complete outage when all worker nodes ran out of disk space simultaneously. This was caused by:

  1. Unbounded log growth from a misconfigured application
  2. No disk space monitoring alerts
  3. Default emptyDir volumes filling up node storage

The Impact

  • All pods became pending as nodes were marked unschedulable
  • 4 hours of downtime during peak traffic
  • Data loss for applications using emptyDir storage

Lessons Learned

  1. Monitor disk usage: Set up alerts at 70%, 80%, and 90% capacity
  2. Log rotation: Configure proper log rotation for all containers
  3. Resource limits: Set storage limits on emptyDir volumes
  4. Multi-zone deployment: Distribute workloads across availability zones

The Fix

We implemented:

By Adam

read more