Home on Kubernetes Fail

About

Tue, 26 May 2026 18:00:00 +0200

About Kubernetes Fail

Our Mission

To create the most comprehensive collection of Kubernetes failure stories, helping operators learn from others’ mistakes and improve cluster reliability.

Who We Are

We’re a team of Kubernetes operators, SREs, and developers who have collectively managed thousands of nodes across multiple clusters. We’ve made (and learned from) every mistake in the book.

Why This Matters

Kubernetes is complex. Failures will happen. What matters is:

Learning from each incident
Sharing knowledge with the community
Improving our systems and processes

Get Involved

Submit your stories: Share your Kubernetes failure experiences
Suggest improvements: Help us make this resource better
Spread the word: Tell others about this resource

Contact

Email: admin@kubernetes.fail GitHub: github.com/kubernetes-fail

Cheatsheet

Tue, 26 May 2026 18:00:00 +0200

Kubernetes Cheatsheet

Common Commands

Cluster Management

kubectl get nodes - List all nodes
kubectl get pods -A - List all pods in all namespaces
kubectl get events -A - View cluster events

Debugging

kubectl describe pod <pod-name> - Get detailed pod information
kubectl logs <pod-name> - View pod logs
kubectl exec -it <pod-name> -- /bin/bash - Execute command in pod

Resource Management

kubectl top nodes - View node resource usage
kubectl top pods - View pod resource usage
kubectl get pvc - List persistent volume claims

Troubleshooting Tips

Node Issues

Check kubelet status: systemctl status kubelet
Check disk space: df -h
Check memory usage: free -h

Pod Issues

Check pod events: kubectl describe pod <pod-name>
Check pod logs: kubectl logs <pod-name> --previous (for previous instance)
Check resource limits: kubectl describe pod <pod-name> | grep -i limit

Network Issues

Test connectivity: kubectl exec <pod-name> -- ping <target>
Check DNS resolution: kubectl exec <pod-name> -- nslookup <service>
Check service endpoints: kubectl get endpoints <service-name>

Commands

Get resources

$ kubectl api-resources --api-group ""
NAME SHORTNAMES APIVERSION NAMESPACED KIND
bindings v1 true Binding
componentstatuses cs v1 false ComponentStatus
configmaps cm v1 true ConfigMap
endpoints ep v1 true Endpoints
events ev v1 true Event
limitranges limits v1 true LimitRange
namespaces ns v1 false Namespace
nodes no v1 false Node
persistentvolumeclaims pvc v1 true PersistentVolumeClaim
persistentvolumes pv v1 false PersistentVolume
pods po v1 true Pod
podtemplates v1 true PodTemplate
replicationcontrollers rc v1 true ReplicationController
resourcequotas quota v1 true ResourceQuota
secrets v1 true Secret
serviceaccounts sa v1 true ServiceAccount
services svc v1 true Service

Namespaces

Namespaces are a way to divide cluster resources between multiple users.

Stories

Tue, 26 May 2026 18:00:00 +0200

Kubernetes Failure Stories

Welcome to our collection of real-world Kubernetes failure stories. These are actual incidents that have occurred in production environments, shared to help others learn and avoid similar issues.

Featured Stories

The Great Disk Space Crisis

Date: March 15, 2023 Environment: Production cluster with 50 worker nodes Impact: 4 hours of complete outage

What Happened: One of our earliest major incidents occurred when a logging sidecar container was misconfigured and began writing unbounded log files to an emptyDir volume. The application team had set the log level to DEBUG during troubleshooting but forgot to revert it. Within 6 hours, all worker nodes ran out of disk space, causing a cascade of failures.

Whoami

Tue, 26 May 2026 18:00:00 +0200

About the Team

Our Background

We are a group of Kubernetes operators, Site Reliability Engineers, and DevOps professionals who have collectively managed thousands of nodes across multiple Kubernetes clusters in production environments.

Our Mission

Our mission is to create the most comprehensive collection of Kubernetes failure stories and lessons learned. We believe that sharing our experiences (and mistakes) helps the entire community improve cluster reliability and operational excellence.

Team Members

Adam - Founder & Lead Operator

8+ years of Kubernetes experience
Managed clusters from 10 to 1000+ nodes
Specializes in networking and security

Sarah - SRE & Reliability Expert

5+ years of site reliability engineering
Expert in monitoring and alerting systems
Focuses on incident response and post-mortem analysis

Michael - DevOps Engineer

6+ years of infrastructure automation
CI/CD pipeline specialist
Focuses on GitOps and infrastructure as code

Our Philosophy

We believe in:

RBAC Misconfiguration Leads to Security Breach

Mon, 15 Apr 2024 09:15:00 +0200

Security Incident

An overly permissive RBAC configuration allowed an attacker to:

Gain access through a compromised developer account
Escalate privileges using cluster-admin binding
Deploy cryptocurrency mining containers
Access sensitive data from other namespaces

How It Happened

Default namespace had cluster-admin role binding
No network policies between namespaces
No pod security policies enforced
Developer accounts had excessive permissions

Immediate Actions

Revoked all compromised credentials
Deleted malicious pods
Isolated affected nodes
Rotated all cluster certificates

Long-term Fixes

Implemented principle of least privilege
Enabled network policies
Deployed pod security admission
Regular RBAC audits
Developer security training

Network Partition Causes Split-Brain

Sun, 14 Apr 2024 14:30:00 +0200

The Incident

A network partition between availability zones caused our etcd cluster to split into two separate quorums, resulting in:

Inconsistent cluster state
Scheduling conflicts
Data corruption in stateful applications

Root Cause

Network provider had a routing issue between zones
Our etcd cluster was spread across 3 zones but had no zone-aware configuration
No network partition detection or automatic fencing

Resolution

Manually identified the healthy partition
Forced etcd to use only the healthy members
Rebuilt the unhealthy nodes from backups
Implemented zone-aware etcd configuration

Prevention

Configure etcd with strict zone awareness
Implement network health checks
Regular disaster recovery drills

First Kubernetes Failure: Node Out of Disk

Sat, 13 Apr 2024 10:00:00 +0200

What Happened

Our production Kubernetes cluster experienced a complete outage when all worker nodes ran out of disk space simultaneously. This was caused by:

Unbounded log growth from a misconfigured application
No disk space monitoring alerts
Default emptyDir volumes filling up node storage

The Impact

All pods became pending as nodes were marked unschedulable
4 hours of downtime during peak traffic
Data loss for applications using emptyDir storage

Lessons Learned

Monitor disk usage: Set up alerts at 70%, 80%, and 90% capacity
Log rotation: Configure proper log rotation for all containers
Resource limits: Set storage limits on emptyDir volumes
Multi-zone deployment: Distribute workloads across availability zones

The Fix

We implemented: