Bot Filtering: Remove Invalid Traffic from your Data
Last Updated:GoogleBot crawls over the internet 24/7, and likely loads your ads multiple times a day. You likely don't want to count impressions or clicks made those bots, crawlers and spiders. Most ad servers, like AdGlare, uses bot filtering at engine level. Bot traffic accounts for more than 50% of all global internet traffic. An incredible number that shouldn't be neglected. Unless you're running in-house campaigns, it's imperative that your ad server filters bot traffic to avoid report discrepancies.
Bots accounting for 51.8% of all Internet TrafficAlthough bot activity fluctuates over the years, we can't deny the huge impact that bots and spiders have on our statistical data. Back in 2016, Incapsula released a great infographic to give us an update on where we're heading. Half of all internet traffic comes from bots.
So, how does Bot Filtering work?Genuine bots and crawlers tell us who they are via the User Agent string that is passed along with each HTTP request. This string will be matched against IAB's list of known bots and spiders to determine if we're dealing with a human or non-human traffic. For example, Googlebot uses the following user agent string:
Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)The Media Rating Council (MRC) has set a standard for detection and filtering of invalid traffic. If the ad serving engines receive a request from such a user agent, an advertisement will be returned but the impression or click will simply not be logged. AdGlare uses this method to make sure page layout remains the same whether a bot or a human visits the page. This is important to let Google determine which content is above the fold - a significant factor in SEO.
Note that we're not speaking about "bad bot" filtering for automated ad exchanges or programmatic ad buying. We're speaking about detecting and filtering spiders and crawlers (as per the infographic above).
Why does this all matter?In the online advertising industry, publishers are getting paid to show advertiser's ads. The publisher's inventory is bought to shows these ads in order to attract potential buyers, which are humans, not bots. As most inventory is sold on a CPM basis, it doesn't make sense to serve half of the campaign to bots while getting paid for it. It's therefore common practice for advertisers to insist on bot filtering when closing a deal with a publisher or ad network.
How to enable Bot Filtering in AdGlareIt's highly recommended to enable bot filtering to minimize discrepancies with third party ad servers, especially if you're a publisher. To do so, follow these steps:
1 Click Settings and then Global Settings
2 Click the Engine Configuration tab
3 Enable the Bot/Crawler Filter
Bot List of 2019
The following bots, crawlers and spiders are filtered by AdGlare:
Analytics SEO Crawler
CSS Certificate Spider
Cc Cc Bot
CloudFlare AMP Fetcher
CloudFlare Always Online
Domain ReAnimator Bot
Facebook External Hit
Genieo Web Filter
Gluten Free Crawler
Gmail Image Proxy
Google PageSpeed Insights
Google Partner Monitoring
Google Structured Data Testing Tool
IIS Site Analysis
Larbin Web Crawler
Lets Encrypt Validation
Netcraft Survey Bot
PHP Server Monitor
RamblerMail Image Proxy
Screaming Frog SEO Spider
Seznam Email Proxy
Site24x7 Website Monitoring
Skype URI Preview
Tiny Tiny RSS
UkrNet Mail Proxy
Visual Site Mapper Crawler
W3C CSS Validator
W3C I18N Checker
W3C Link Checker
W3C Markup Validation Service
W3C MobileOK Checker
W3C Unified Validator
Willow Internet Crawler
Yahoo Cache System
Yahoo Link Preview
Yottaa Site Monitor
Filtering IP addresses from Malicious NetworksIn addition to filtering invalid traffic from bots, you may also want to consider to filter requests made from known malicious networks. A quick search on Google can provide you with a list of IP addresses (likely CIDR notations) from networks known to be infected with software to automatically crawl pages to artificially inflate impressions. AdGlare can filter those impressions and clicks at two levels:
- Campaign level. See the Targeting Rules tab when editing a campaign.
- Engine level. See the page Settings => Main Configuration.
That's not all. A few more things...Now you're right on track to improve your CTR and the quality of your inventory, it's absolutely worth it to consider the following practices as well.
- Lazy-Loading Ads. A banner is only loaded when it's scrolled into view, right in front of the visitor's eyes. Highly recommended, as it doesn't make sense to load ads below the fold that are never seen.
- Nofollow attributes. This avoids passing on the link juice to low authority domains on outbound links. Buying links is a black hat SEO technique, penalized by Google.
Do It Yourself: Creating a Bot TrapIf you have some time to spare, you can create a simple bot trap. The idea? Create a small 10 x 10 pixel transparent image that links to a secret URL (i.e. your trap page). The image is invisible to humans, but bots will follow the URL and end up on your secret page. Simply log how many times your secret page is visited and match that against the total number of visits. You'll then have a good guess of the amount of bot traffic that your website receives.
Permalink for Bloggers
This article stays online on the following URL: