Scraping millions of pages a day: what actually breaks

Short summary

A veteran scraping engineer shares hard-won lessons from running a platform processing millions of pages daily at 95% extraction success. Key failures include unbounded queues causing OOM, retry storms acting as accidental DDoS, URL duplication wasting fetch budget, silent parser drift going unnoticed, and inability to debug past failures. Solutions include bounded queues, full-jitter backoff with circuit breakers, canonicalization at enqueue time, field-level fill-rate monitoring, multi-strategy extraction, and archived raw HTML for replay debugging.

• •Bounded queues with backpressure prevent OOM crashes from bursty crawl patterns
• •Full-jitter backoff and per-domain circuit breakers stop retry storms from DDoSing target sites
• •Field-level fill-rate monitoring and multi-strategy extraction catch silent parser drift before downstream consumers notice

Generated with AI, which can make mistakes.