Cloudflare draws the line: from Sept 15, AI can't scrape your ad pages for free
On July 1, Cloudflare dropped a bomb — a new deadline for the entire AI industry. Starting September 15, 2026, the default settings for new Cloudflare customers, newly created sites, and all free-tier customers will block "mixed-use" crawlers from pages that host ads.
What's a mixed-use crawler? One that does search, runs agents, and harvests training data at once (yes, they named the big search engines' bots). Under the new rule, to crawl an ad page, the site owner has to manually opt in.
Cloudflare CEO Matthew Prince: most internet traffic is no longer human, so we have to move faster for a sustainable ecosystem to emerge.
Publishers finally get weapons
This isn't just blocking — it ships a control panel:
- Search, Agent, and Training bots are managed separately, no more lumped together
- Publishers can see exactly which AI company is crawling and what
- They can also see whether those AI bots actually send traffic back — none? Block them, or charge
The companion play is called Pay Per Use, an upgrade to the earlier Pay Per Crawl — you don't just charge when scraped, you charge when your content creates value (shows up in answers, feeds training). Launch partners are Ceramic.ai and You.com.
Why this matters
Cloudflare's data says over 50% of AI crawler traffic is spent re-fetching unchanged pages — pure bandwidth waste. For publishers, the old deal was brutal: feed AI for free, burn your own server cost, and the crawler might not even send you traffic.
For anyone running a content site, this policy hits directly: who scrapes your articles, what they do with them, whether you get paid. Sites created after Sept 15 default to your side.
Google, naturally, pushed back, pointing to Google Extended as an opt-out for training. But Cloudflare's point stands: your search dominance rides on AI training as a bundled cost, and that bill is now due.
Plainly, the era of free data harvesting is what Cloudflare wants to end for publishers. Whether AI companies pay up quietly — September 15 will tell.