Australian publisher Man of Many is welcoming AI crawlers instead of blocking them and logging 1.3 million crawler visits a week, with AI referral traffic climbing fast. For most businesses whose content is marketing rather than product, blocking crawlers removes them from the answers AI assistants write.
The Take: Blocking AI crawlers is not protecting your content. For most Australian businesses it is quietly deleting you from the places buyers now go to ask what to buy. The publishers treating crawler traffic as a threat are guarding a door while their customers walk in through a window.
What changed: Australian lifestyle publisher Man of Many opened its site to AI crawlers instead of walling them off, and now logs 1.3 million AI crawler visits a week with referral traffic from AI platforms climbing fast. Referrals from Claude are up 594% and from Gemini 539% over the period, while Google, Bing and Amazon run the busiest crawlers. The publisher treats the crawlers as a distribution channel rather than a threat, and the referral numbers are following the decision.
Growth in referral traffic Man of Many saw from Claude after opening its site to AI crawlers
The catch: Blocking is not free. When you close the door to a crawler you also close the door to being quoted, cited and sent traffic when someone asks an AI assistant for a recommendation. A wall that keeps the machine out keeps your business out of the answer it writes. More buyers now start a purchase by asking an assistant, and the brands that get named are the ones the model was allowed to read in the first place.
The local angle: Big Australian media owners like Nine have leaned toward restricting crawlers to protect the value of their reporting, which makes sense when the content is the product you sell. For a plumber, a law firm or an online retailer the maths runs the other way. Your words are marketing, not the product, so being read by the machine that recommends you is the whole point. Blocking to protect pages you already give away for free costs you the reach and keeps none of the value.
For Australian operators: