1:09The IAB spiders and bots list
Mistake. Counting a search crawler as a reader, and billing an advertiser.
Lesson notes
Mistake. Counting a search crawler as a reader, and billing an advertiser.
The IAB and ABC Spiders and Bots List stops that. It’s a shared list of known robots.
Early counters logged crawlers too, and auditors kept different bot lists, so counts disagreed. The industry merged them into one list, now run by IAB Tech Lab and updated monthly. Two files. One lists real browser user agents, the name tag every visitor sends. One lists known robots. Each visit must match a real browser, and avoid the robot list. Fail either, it’s filtered. That covers general invalid traffic, or GIVT. The limit? A bot that lies about its name gets through. That’s a different fight.
Your check. Google Analytics already drops bots on this list. If your server logs show far more pageviews, robots are a likely reason.
Next: agentic ad protocols, ARTF and AAMP.
Go deeper · 3 min readGoogle Adsense Bots: What You Need to KnowUp next in this series
1:21Agentic ad protocols: ARTF and AAMPAgentic ad protocols: ARTF and AAMP
1:10CoMP: getting paid when AI crawls your contentCoMP: getting paid when AI crawls your content
1:25OpenRTB started with block listsOpenRTB started with block lists
1:18Header bidding, in one storyHeader bidding, in one story
1:24Who actually handled this impression? (schain)Who actually handled this impression? (schain)
1:05Inside an OpenRTB bid requestInside an OpenRTB bid request
1:01buyers.json and DemandChain: the buyer-side mirrorbuyers.json and DemandChain: the buyer-side mirror
1:05ads.cert: signing the bid requestads.cert: signing the bid request
1:07OM SDK: how viewability gets measured in appsOM SDK: how viewability gets measured in apps
1:05SHARC: the standard meant to replace SafeFrame and MRAIDSHARC: the standard meant to replace SafeFrame and MRAID
1:08DSA transparency in the EU bid streamDSA transparency in the EU bid stream