Managing bots for better security and Web experience

Because of the unprecedented level of contact that online businesses have with their customers and end-users via business websites, a lot of business information and processes that can be easily accessed, have found their way onto these websites.

According to Akamai’s Product Marketing Manager in Asia Pacific & Japan, Sudeep Charles, this is often enabled with automated tools known as ‘bots’.

Sudeep said, “For many organisations, bots represent up to 40-percent of their overall site traffic, from good bots engaged in essential business tasks to bad bots performing harmful activities.

“The majority of bot traffic for online retail sites, for example, can be attributed to bots that either have a legitimate business role, including search engines and site development & monitoring services, or perform non-malicious third-party activities such as web archiving.”

He did caution however that the vast majority of bot traffic for event ticketing websites, come from unknown bots that do not have recognised business purpose.

Besides hidden motivations that are likely dubious, bot traffic can also reduce site performance for users and ultimately increase IT cost.

Managing bots

Sudeep shared that there are a few solutions like Akamai’s Bot Manager, but none that have the intelligence Akamai has in terms of scale and global presence. He said, “Bot Manager, has been well-received globally as well as in Asia Pacific and Japan.”

Basically, bot management can help businesses improve a user’s web experience by reducing bot impact on the Web infrastructure, prevent price and content scraping, as well as reduce IT costs by offloading operational overhead required to identify and manage bots to Akamai.

From a technical point of view, organisations can benefit by gaining visibility into the characteristics and amount of bot traffic accessing their site. This includes detecting unknown bots in real-time using evolving detection methods devised and updated by Akamai’s threat research team.

Also, because the solution is able to employ advanced strategies to better manage interactions with different bots based on business policy and is able to leverage Akamai’s visibility into bot traffic (there are over 1,200 Akamai-categorised bots).

Good to know information

Sudeep said, “If you look at the range of solutions on the market today for bot management – whether it’s a standalone solution, a checkbox on a Web application firewall (WAF) solution, or even something organisations are doing themselves, all of these solutions have one thing in common – they block bots.”

Blocking bots is a traditional security approach addresses the symptoms of bots, he said adding that, “What we’ve learned is that the bot problem requires a different approach to actually address the challenge.

“The fact that is forgotten is that a bot is just a remotely operated script to get something from a site. When you block the bot, while you prevent the bot from getting what it came for, you’ve also alerted the bot operator that you’ve detected them.”

This actually alerts the bot operator to just modify the bot to evade detection, and a  more sophisticated bot operator would probably change the behaviour or the characteristics of its requests to better mimic the businesses’ legitimate users.