The crawl-to-referral ratio measures how many pages a bot reads for every visitor it sends back. For traditional search crawlers, that ratio sits near 1:1 — a fair trade.

AI training bots broke that bargain. One analysis found Anthropic's crawler read thousands of pages for every single referral returned, consuming bandwidth while giving nothing back. Publishers noticed. Robots.txt — the file at a site's root that tells bots what they may read — became a battleground. Cloudflare data shows bots now account for more than half of all web traffic, and by late 2026 Cloudflare began blocking AI crawlers by default on new domains rather than merely asking them to leave.

The core tension is that robots.txt is a request, not a lock. A crawler that ignores it faces no automatic penalty — which is why the fight has moved from text files to network-layer blocks, licensing deals, and lawsuits.