Bad bots are big news largely because of the FBI investigation into Russia’s involvement in the 2016 presidential election. But bad bots are a bigger problem than automated tweeting: 42.2% of all website traffic comes from bots; and 21.8% of it is down to bad bots.
Distil Networks’ 2018 Bad Bot Report, based on an analysis of hundreds of billions of bad bot requests, shows that bad bot traffic increased by 9.5% in 2017. Bad bots differ from good bots, whose traffic also increased by 8.8% to 20.4%. It means that only — on average — 57.8% of visiting traffic comes from a genuine human being interested in the website content.
Good bots are those that all websites require. They include the search engine page indexing bots from Google and Bing, and they bring humans to the site. Bad bots, however, are secretive and nefarious. They come from outright criminals and commercial competitors; and their purpose is to detract and/or steal from the website.
Distil highlights eight different bad bot functions: price scraping, content scraping, account takeover, account creation, credit card fraud, denial of service, gift card balance checking, and denial of inventory. They fall into three primary categories: competitive, organized criminal, and nuisance.
Price scraping and content scraping are generally competitor attacks. Price scraping allows competitors to maintain price levels slightly lower to score more highly in search engine rankings. Content scraping is simply the theft of proprietary content to augment another site’s own content.
Account takeover bots are automated attempts at illegal log-ins. They can deliver brute-force attacks cycling through the most popular passwords to see if one of them works, or they can use the process known as credential stuffing.
Distil reports a 300% increase in credential stuffing bad bots in the weeks following a new major credential theft. This involves the automatic application of stolen passwords on different websites. “Here,” notes the report, “bot operators make two assumptions. The first is that people reuse their credentials on many websites. The second is that newly stolen credentials are more likely to still be active. This is why businesses should anticipate bad bots running those credentials against their website after every breach.”
Account creation bad bots simply generate vast numbers of new accounts — for example, on Twitter — to spam out messages or amplify propaganda.
Credit card fraud bots test out credit card numbers, trying to identify missing information — such as the expiry date and the CVV.
The denial of service bad bot can be either competitive or nuisance. It can be used to reduce the performance of a competitor, or to disrupt the service of a small website either because of a grudge, or simply because it is possible. It can be effected either from a small number of attacking IP addresses, or from a larger number of rotating addresses. Automated defenses often fail because the number of accesses from each IP address is below the warning threshold before it moves to other addresses, while manual whack-a-mole IP blocking simply cannot keep up.
Gift card balance checking bots are used to steal money from gift card accounts that contain a balance.
‘Denial of inventory’ is a relatively new competitor attack prompted by the growth of ecommerce. In this attack, bots place stock items in online shopping baskets, taking them out stock lists. If the item is no longer available, then visiting human buyers will go elsewhere to make the purchase.
Bad bots are a difficult problem. Many website owners are not aware of them, while common defenses have little effect. Geo-blocking, for example, is only somewhat effective. Many sites block all Russian traffic. While this will inevitably include some bad bot traffic, it may also exclude some genuine human traffic. Russia is, however, the most blocked country.
In reality, the greatest source of bad bot traffic is the U.S. (although the operators may be elsewhere). According to Distil, 45.2% of all bad bot traffic originates in the United States (China is second, but way down with just 10.5%). This is because nobody, anywhere in the world, is likely to block all traffic coming from the U.S.
“This year bots took over public conversation, as the FBI continues its investigation into Russia’s involvement in the 2016 U.S. presidential election and new legislation made way for stricter regulations,” said Tiffany Olson Jones, CEO of Distil Networks. “Yet, as awareness grows, bot traffic and sophistication continue to escalate at an alarming rate. Despite bad bot awareness being at an all-time high, this year’s Bad Bot Report illustrates that no industry is immune to automated threats and constant vigilance is required in order to thwart attacks of this kind.”