Great Stack to Doesn't Work #10 — Season Finale: "When PagerDuty Calls at 3 AM"

Short summary

Three production incidents root-caused to disabled safety mechanisms (PostgreSQL fencing), config validation gaps (Kubernetes trailing space in hostname), and monitoring blind spots (Redis failover during cache invalidation). Lessons span automatic failure isolation (STONITH), CI/CD gates for config changes, and monitoring for impossible states like negative replication lag.

• •PostgreSQL split-brain from disabled Patroni fencing created 8 min of conflicting writes requiring 4 hrs manual reconciliation
• •Kubernetes ConfigMap typo (trailing space in hostname) caused 32-min platform outage via invisible DNS failure
• •Redis failover during cache invalidation timeout led to stale pricing for customers

Generated with AI, which can make mistakes.