The HTTP Code Your AI Agent Doesn't Handle Yet: 402

Short summary

Cloudflare's Pay-Per-Crawl (July 2025) returns HTTP 402 with pricing in headers, requiring crawlers to choose between free fallbacks, per-page ceilings, and total spend tracking. A naive agent treating 402 like a free request spent 10x over budget; proper handling keeps crawlers cost-efficient. Code example and decision trees are grounded in 2,190 production crawler runs.

• •HTTP 402 now requires explicit policy: free alternatives, per-page budgets, and run-level spend caps
• •Naive 'just pay' agents can overspend 10x; structured decision trees prevent runaway costs
• •Working Python code and validated patterns from 2,190 real-world crawler runs

Generated with AI, which can make mistakes.