Cloudflare will block mixed-use AI crawlers by default from 15 September and let publishers charge them per crawl. If you make content, you just got a lever you did not have last month.
For years the deal was simple. You made the content, the crawlers took it and search sent some traffic back. That deal is being rewritten in front of us.
Cloudflare has decided AI companies should pay to read the open web, and from 15 September it will start blocking them by default.
Cloudflare sits in front of a large share of the world's websites, so when it changes a default, the change carries. Its pay per crawl system lets a publisher do one of three things with each AI crawler that shows up: let it in for free, charge a fee per crawl or block it outright. From 15 September 2026 the default flips. New Cloudflare customers, new sites set up by existing customers and all existing free customers will block "mixed-use" crawlers on any page that carries ads, unless the owner says otherwise.
The company is already stretching the idea from pay per crawl to "pay per use", so publishers get paid when their content creates value inside an AI product, not only when a bot fetches it. Early partners include Ceramic.ai and You.com. Big names like Conde Nast, TIME, the Associated Press and Fortune have backed the default-block stance.
Why it matters
Australian businesses that built their acquisition on ranking and getting clicks have watched AI answers keep the visit and send nothing back. Cloudflare is the first infrastructure player big enough to charge for the raw material. If you publish content, you now have a lever you did not have last month. If your own product leans on scraping third-party content to work, your costs are about to change.
The date Cloudflare starts blocking mixed-use AI crawlers by default on ad-carrying pages
What to do about it
The free-for-all era of the open web is ending, and the businesses that treat their content like inventory will be the ones setting the price.