The 2026 AI-Crawler Reality Check: What the Data Says for Developers

Short summary

A data-driven analysis of AI crawler behavior in 2025-2026 reveals that no major AI crawler except Google executes JavaScript, meaning client-side-only content is invisible to most AI engines. AI crawler volume is up 305% but crawl-to-referral ratios are extremely lopsided (38,065:1 for Anthropic). The practical takeaway: server-render content, use semantic HTML, and treat AI visibility as a technical engineering discipline overlapping with classic SEO.

• •No major AI crawler except Google executes JavaScript — SSR/SSG is essential for AI visibility
• •AI crawler volume up 305% YoY but referrals remain under 1% of web traffic
• •llms.txt adoption grew 8.8x but 97% of files received zero requests — use robots.txt instead
• •Schema/JSON-LD does not independently predict AI citation once content quality is controlled for

Generated with AI, which can make mistakes.