Cloudflare blocks AI crawlers by default, letting sites choose what gets scraped

Cloudflare is now the first major internet infrastructure company to block AI crawlers by default when they try to access website content without permission or payment. Starting today, website owners can choose whether to allow AI crawlers and set rules for how their content is used.

“Original content is what makes the Internet one of the greatest inventions in the last century, and it’s essential that creators continue making it. AI crawlers have been scraping content without limits. Our goal is to put the power back in the hands of creators, while still helping AI companies innovate. This is about safeguarding the future of a free and vibrant Internet with a new model that works for everyone,” said Matthew Prince, CEO of Cloudflare.

AI companies can also identify whether their crawlers are used for training, inference, or search, giving site owners more information to make decisions. Cloudflare says this new default is a step toward a fairer system for both content creators and AI developers.

For years, the internet ran on a simple system: search engines scanned websites, sent users to those sites, and in return, creators earned traffic and ad revenue. This helped people find useful information and gave creators a reason to keep publishing good content. But that system is breaking down. AI crawlers now collect text, images, and articles to generate answers without linking back to the original source. Creators lose both income and the basic feedback of knowing someone saw their work. If there’s no reason to keep creating quality content, everyone loses, and the open web as we know it is in danger.

“AI companies, search engines, researchers, and anyone else crawling sites have to be who they say they are. And any platform on the web should have a say in who is taking their content for what,” said Steve Huffman, CEO of Reddit. “The whole ecosystem of creators, platforms, web users and crawlers will be better when crawling is more transparent and controlled, and Cloudflare’s efforts are a step in the right direction for everyone.”