EBS gp2 burst credits ran dry and our builds slowed to a crawl

Short summary

Buildkite's build agents experienced afternoon slowdowns traced to EBS gp2 volumes exhausting burst credits after hours of small random writes. Migrating to gp3 (flat 3000 IOPS baseline, no credit bucket) restored p95 from 9.2 to 4.1 minutes and cut costs 20%. Post includes AWS CLI migration path and CloudWatch monitoring.

• •EBS gp2 volumes drain burst credits during sustained load, hard-capping IOPS at baseline until recovery
• •gp3 provides flat 3000 IOPS baseline with no burst credit dance, also 20% cheaper for this workload
• •Migration is one-line volume modification with no downtime; update launch templates for new instances

Generated with AI, which can make mistakes.