The AI Crawler Census: 14 Bots, 97,454 Requests, and More Knocks Than Every Search Engine Combined
We logged 1,348,706 requests across two production web properties for 174 days and classified every user-agent. AI crawlers now generate 7.2% of all observed traffic — more than Googlebot, Bingbot, DuckDuckBot, Baiduspider, and PetalBot combined. OpenAI alone accounts for 38% of the AI crawl, 69% of AI requests still feed training pipelines, and a robots.txt honeypot shows 99% of AI-crawler violations trace to a single bot. For publishers, infrastructure teams, and anyone deciding what to do about the new tenants of the crawl economy.