Cloudflare Will Filter Out Web Crawlers That Serve AI Companies
The hosting platform wants sites to have more control over how AI companies use their content.
Cloudflare has announced plans to automatically block mixed-use web crawlers that index websites for search engines and act as AI agents and trainers at the same time. The company previously offered its customers the optional ability to prevent crawlers from scraping their sites for AI chatbots, but now Cloudflare’s stance is becoming more defensive by default.
“Now that the majority of traffic on the Internet is non-human, we must go further and act faster so that a sustainable ecosystem can emerge,” Matthew Prince, Cloudflare’s CEO and co-founder shared in a statement. “Cloudflare’s new tools and partnerships give website owners increased visibility and commercial opportunities and benefit AI companies that have bots with clear and transparent intent. We hope that our proposed default changes encourage mixed use crawlers to separate out search from agent use and training.”
Web traffic used to indicate that people were viewing a website’s ads or paying for its subscriptions, but the popularity of AI models that can visit sites on a user’s behalf to pull up-to-date information has upended that system. Cloudflare’s new approach is an attempt to rebalance the relationship in a way that’s fair for both AI companies and anyone running a website.
Starting September 15, 2026, new customers and new websites from existing Cloudflare subscribers will default “to allow for search but block training and agent use for pages with ads.” Mixed-use crawlers that don’t give site owners the option to choose whether their site is used for AI will also be blocked on pages with ads by default. Users with free accounts will also switch to these defaults unless they opt-out ahead of the September 15 deadline, according to the company.
As part of these changes, Cloudflare is also releasing a new version of the Pay Per Crawl feature it introduced in 2025 that allowed websites to block AI web crawlers by default unless companies paid to scrape their content. The feature is now called Pay Per Use, and rather than base payments on whether a webpage has been crawled, Cloudflare says site owners will be paid when their content appears in answers from AI chatbots. The announcement only mentions partnerships with Ceramic.AI and You.com, but Cloudflare likely hopes other AI companies will join as its customers opt in.
Besides generally trying to make the relationship between websites and AI companies more fair, as TechCrunch notes, Cloudflare also seems to be indirectly targeting Google. The company’s announcement mentions that “the largest search engine has access to about 2X more information than leading AI companies because they make it difficult for customers to remain discoverable without also being used for AI.” Google’s main crawler, Googlebot, both indexes websites for the company’s various search engines and collects information to train Gemini and power AI features like AI Overviews and AI Mode. Google lets websites opt-in to a separate crawler called Google-Extended that only crawls websites for traditional search results, but if a publisher wanted to be included in AI Mode results, but doesn’t want their content to train Google’s models, they don’t have an option. Cloudflare’s new policy is an attempt to force Google and other companies with mixed-use crawlers to change their tactics.
