Exponential backoff with jitter stopped our CI retry storms

Short summary

A 40-second Buildkite metadata outage stretched to 6 minutes because CI agents retried in lockstep, creating a retry storm that prevented recovery. Full jitter adds randomness to exponential backoff, decorrelating retries across time; combined with a retry budget (capping retries at 20% of traffic) and a circuit breaker (blocking requests during recovery), the herd no longer suffocates a recovering service. The pattern trades per-request latency predictability for system-wide stability.

• •Synchronized retries from thousands of clients create retry storms that extend outages
• •Exponential backoff with full jitter decorrelates retry timing across the herd
• •Retry budget + circuit breaker provide hard caps on retry volume and guaranteed quiet windows

Generated with AI, which can make mistakes.