Microsoft Clarity's Bot Analytics dashboard now flags bots requesting URLs your robots.txt disallows, with trends and filters by bot and operator. As AI crawlers multiply, knowing who ignores the rules is becoming a real signal. Here is why it matters.
Robots.txt is a sign on the door, not a lock. Now you can at least see who walked straight past it.
Microsoft Clarity has upgraded its Bot Analytics dashboard. It now detects when a bot requests a URL your robots.txt has told it to leave alone, and lets you filter the offenders by bot and by operator, with trends over time.
Robots.txt is a polite request, not a wall. A well-behaved crawler reads it and obeys. A badly-behaved one reads it and carries on. Until now most site owners had no easy way to tell which bots were ignoring the instruction. Clarity, which is free, just made that visible.
The timing is not an accident. The number of AI crawlers scraping the web for training data and live answers has exploded, and not all of them play by the old rules.
Why it matters
For Australian businesses this is two things at once. It is a security and cost signal, because bot traffic that ignores your rules can chew bandwidth and skew your data. It is also an AI visibility signal, because knowing which AI crawlers are reading your site tells you who might be citing you in their answers.
Microsoft Clarity is free, which means this bot visibility now sits within reach of any Australian business, not just the ones with enterprise tooling.
The broader point is that machines are now a serious share of your traffic, and most businesses cannot see them clearly. You cannot manage what you cannot measure.
What to do about it
The web is more machine than it has ever been. Seeing the machines clearly is the first step to deciding which ones you want.