Reuters and Time have switched to a block-all default for AI crawlers, only letting bots through a whitelist. It is a sharper stance on who gets to train on or retrieve their content, and a sign the value exchange is being renegotiated.
If your content is worth training on, it is worth being asked for. The default just flipped from take to ask.
Two major publishers have flipped their default on AI crawlers from allow to deny. Reuters has moved from default allow-all to default disallow-all, letting bots through only by exception. Time now runs a whitelist of around 70 bots. The message to AI companies is simple. If you want the content, have a conversation first.
Reuters spent two years watching bot activity before making the call, gathering enough data to know which crawlers it could block without hurting revenue. It already has licensing deals with the likes of Microsoft and Meta. The point is leverage. Make access something you grant, not something taken by default.
Of AI bot scrapes ignored robots.txt permissions in late 2025, which is why publishers are moving past polite requests. Source: Tollbit via Digiday, June 2026.
Why it matters
This is the value exchange being renegotiated in real time. For years, content was scraped freely to train models that now answer questions without sending a click back. Big publishers are drawing a line. The trouble is that polite blocking does not always work, with nearly a third of AI scrapes ignoring robots.txt rules.
For Australian businesses, this is not just a publisher fight. Every brand with a content library is making the same quiet decision. Block the crawlers and you protect the content but risk vanishing from AI answers. Allow them and you feed the machine that may replace your traffic. There is no clean answer, only a trade-off to make on purpose.
What to do about it
The free buffet is closing. Choose who eats from your content rather than letting anyone with a crawler help themselves.