Cloudflare Blocks AI Bots, Pitches ‘Pay Per Crawl’ to Protect Online Content

Cloudflare, the internet infrastructure firm that hosts about 20% of the web, has launched new tools allowing websites to block unauthorized AI bots from scraping their content. The system, already active on a million sites, is designed to give content creators more control and eventually allow them to charge AI firms for access.

The move comes amid rising concern that AI companies are using web content without permission or payment. Prominent publishers like Sky News, Buzzfeed, and The Associated Press – along with artists and writers – have voiced opposition to this practice.

Cloudflare’s tools target AI crawlers – automated bots that index websites to train AI systems. Unlike traditional search engine crawlers, which direct users to original sources, AI bots often collect content without attribution or traffic return, undermining publishers’ revenue models.

“This is a game-changer,” said Condé Nast CEO Roger Lynch. “It’s a critical step toward protecting creators and quality journalism.”

The firm is also developing a “Pay Per Crawl” system, enabling websites to charge AI companies for content access. Cloudflare argues that AI firms have broken the “unwritten agreement” between publishers and search engines by using data without fair exchange.

AI bot traffic has surged, now generating over 50 billion requests per day to Cloudflare’s network. Some crawlers also ignore existing exclusion protocols. In response, Cloudflare previously redirected the worst offenders into a “labyrinth” of junk content to waste their resources.

Legal battles are intensifying on both sides of the Atlantic. In the U.K., disputes over AI and copyright have drawn figures like Sir Elton John into the debate. The BBC recently threatened legal action against U.S. AI firm Perplexity for unauthorized content use.

Experts say Cloudflare’s effort is a welcome step, but broader legal frameworks are still needed. “This is a sticking plaster when what’s needed is major surgery,” said Ed Newton-Rex, founder of Fairly Trained. “Real protection requires the law.”

Baroness Beeban Kidron praised the move, calling it “decisive action” from a major tech player. “If we want a vibrant public sphere, AI companies must contribute – financially and ethically – to the communities whose work they rely on,” she said.