Webhook Retries Aren't Optional: The 6-Hour Silent Failure That Changed How We Build Them

Short summary

A postmortem of a six-hour silent webhook delivery failure reveals key engineering lessons: fire-and-forget publishers break catastrophically under real-world partner flakiness. The team implemented exponential backoff with jitter (10s to 30min ceiling, 6 attempts), a durable dead-letter queue with replay tooling, idempotency UUIDs for consumers, and intelligent failure classification to avoid retrying malformed payloads.

• •Silent 6-hour webhook failure caused by missing retry logic after a partner's TLS cert rotation
• •Implemented backoff with jitter, DLQ with replay tooling, and idempotency UUID contracts
• •Key insight: the boring parts (replay tooling, alerting thresholds, partner contracts) prevent the next outage

Generated with AI, which can make mistakes.