The Bot Problem in Digital Advertising Is Getting Worse. IAB Tech Lab Has a Framework.
The IAB Tech Lab has released new guidance on managing AI system bots and crawlers, open for public comment until June 26. The document is aimed at publishers,
Content owners need to understand how their content is being used and what they're receiving in return from non-human traffic.
The IAB Tech Lab has released new guidance on managing AI system bots and crawlers, open for public comment until June 26. The document is aimed at publishers, but the question it asks applies to every advertiser buying digital media.
The guidance asks: how much does it cost you to deliver content to bots, and what are you getting in return? For most publishers, the answer is: a lot, and nothing. Non-human traffic now dominates most of the open web. AI crawlers from every major platform are scraping content at scale. The analytics most teams rely on were not built to draw a clear line between a human pageview and a bot crawl.
The new document builds on IAB Tech Lab's CoMP API work and gives content owners a structured approach for deciding how they allow, restrict or monetise access from non-human user agents. It is the most substantive public treatment of the commercial implications of bot traffic the industry has produced.
Why it matters
If you are an advertiser buying programmatically across the open web, the bot traffic problem directly affects your media quality. Impressions served to bots are not seen by humans. Most brand safety and viewability tools catch some of this. None of them catch all of it. The problem is getting worse as AI crawlers become more sophisticated and harder to distinguish from legitimate browser sessions.
What to do about it
Ask your media agency or DSP for their invalid traffic filtering methodology and the most recent audit result. If they cannot produce a clear answer, that is itself information. For programmatic buys across the open web, require pre-bid IVT filtering using GIVT and SIVT standards as a minimum. For premium placements, direct deals with publishers who publish their traffic audit data give you more confidence in what you are actually buying. The IAB Tech Lab guidance finalises in late June and is worth reading for any team with significant programmatic investment.