Bot Access Control Tool

Created at:
Avatar
Updated

 Introduction

Most website owners are not aware of the amount of Bots visiting their site. This is because Bots usually do not run JavaScript, and thus do not show up on website analytics dashboards (such as Google Analytics). 50% of any average website’s traffic is generated by Bots, traffic that is virtually invisible to analytics tools. Incapsula monitors all website Visits, including Humans and Bots, and provides tools to monitor and control the Bots that are visiting your website.

The Bot Access Rule allows you to control which Bots can access your web site. Incapsula classification includes three types of Bots:

1)    Good Bots: these are Bots (such as Google’s search Bots or Pingdom) which operated by well-known and commonly used services. Incapsula’s client classification engine identifies these Bots and allows them to access your site.

2)    Bad Bots: these include Comment Spammers, SQL Injection Worms and Vulnerability Scanners, as well as other malicious Bots that are positively identified by Incapsula.

3)    Suspected Bots: In many cases, these Bots are used by service providers that have not yet been classified by our classification engine. In other cases, these could be browsers behind proxies or obfuscation devices, which were misclassified by Incapsula.

What will Incapsula do with Good Bots?

All Good Bots are allowed to access your site. If you would like to prevent a certain Bot from accessing your site, you can exclude this Bot from the Good Bot list in the Bot Access rule settings and add it to the Bad Bots list.

What will Incapsula do with Bad Bots?

All Bad Bots are denied access to your site. If you would like to whitelist a certain Bot (for example, a scanner you are using) you can do so via the Bot Access Rule >> Add exception option.

What will Incapsula do with Suspected Bots?

Suspected bots are allowed by default. You may enable the option to require an additional CAPTCHA security check for any Suspected Bot, so only Human Visitors can pass. Through the Events screen in your site dashboard, you can review any event in which a Visitor was required to complete a CAPTCHA. Incapsula shows which clients successfully completed the CAPTCHA (Humans) and which clients failed (Bots) and allows adding client to whitelists / blacklists from the Events page.

 

Guide

 

 1.gif

All Good Bots (like Google and Pingdom) will be allowed to access your site:

  • Bots listed here will not be blocked (Checked by default).
  • “Good Bots” list consists of Bots that are considered beneficial 
  • If you wish to block specific Bots, click “Good Bots…” and uncheck a Bot that you would like to block from accessing your site.
  • Users cannot disable this feature directly, but can manually remove all Bots from “Good Bots” list
 2.gif

Block Bad Bots (like comment spammers and scanners) known to Incapsula:

  • Bots listed here will be blocked (Checked by default).
  • “Bad Bots” list consists of Bots that are considered malicious. 
  • If you wish to block a specific Bot, click “Also block…” and enter the name of a Bot that you would like to block from accessing your site.
  • Users can disable this feature, reverting to “Alert Only”.
 3.gif

Require all other suspected bots to pass a CAPTCHA test:

  • “Suspected Bots” may be presented with a CAPTCHA challenge (Unchecked by default).
  • “Suspected Bots” are: “Unknown Bots”; “Good Bots” that were removed from the “Good Bot” list by the user; and Visitors that are Bots, but are not on the “Bad Bot” list.
  • CAPTCHA challenge will not be given to Visitors that match rules set by the user under “Add exception”.
  • Users can disable this feature, reverting to “Alert Only”

 

Add exception:

  • Users can add exception rules based on: URL, Visitor, IP, Country, User Agent, or a specific Parameter
  • Exception rules will override all other “Bot Access Control” rules. 
For additional information, see also How to set a security exception?
 4.gif

 

Was this article helpful?
8 out of 9 found this helpful
Have more questions? Submit a request

Comments

  • Avatar
    prodport

    I'm guessing the items that show on the "Good Bots" list are common to everyone, like a master list,  and not just our own site, correct?

  • Avatar
    raldridge

    That's correct. 

  • Avatar
    Samuel Aguilera

    I'm trying to block a bad bot using the "Also block" option, but when clicking the "Add" button after typing the bot name nothing happens :(

    Tested in latest Chrome and Firefox.

    Any help would be appreciated.

  • Avatar
    adrian

    To me something is not working here*;

    *

    All Good Bots (like Google and Pingdom) will be allowed to access your site

    • If you wish to block specific Bots, click “Good Bots…” and uncheck a Bot that you would like to block from accessing your site
    • Users cannot disable this feature directly, but can manually remove all Bots from “Good Bots” list

    1.- I do have unchecked this; OpenIndex Bot (Search Bot)

    2.- Even so;

    149.126.78.1      /robots.txt      2/15/13 10:48 PM      425      success 200      GET      HTTP/1.0           Mozilla/5.0 (compatible; OpenindexSpider; +http://www.openindex.io/en/webmasters/spider.html)

    A line of log from my shared hosting account.

    Please respond to this!

    For wphost, I want to suggest, you have to go to; http://www.botopedia.org/

    And verified the bad bot you one to block is in the list in Botopedia.org. if not in that list, you will not be able to block it.

    Very strange and should not happen that way. We should be able to block, for example;

    Deepnet|panscient|Hotbar|MyIE|torrentland|warebay|catchbot -- Something simple like "MSIE 6.0"

    The team in the Incapsula think they can work fast enough to add BOT, I think they are wrong.

  • Avatar
    echai

    Hi Adrian,

    First, thanks for your comment. I will try to clarify…

    Incapsula classifies bots into one of three categories: good, bad and suspected When a bot is in the good bot list we will make sure that it is blocked by rules that intercept automated access (like the Suspected Bots rule or DDoS rules) When a bot is in the bad bots list it will be blocked regardless of other rules When a bot is suspected it means we have not classified it yet as good or bad. If you would like to block such bots you can either add them to the bad bots list or you can treat them by enabling the Suspect Bots rule which basically prevents any automated access to your website.

    When you tried to prevent access from OpenIndex you followed our instructions to remove it from the Good bots list however, this is not enough. You also need to add this bot to the bad bots list by its visitor type if it is recognized by Incapsula like OpenIndex. The instructions in the entry are partial and I have fixed them so other users will not be mislead

    Regarding unclassified bots. Our security team adds new bots to our directory on a daily basis and you are more than welcome to send requests through our support team. However, you can still prevent access from certain bots by enabling the suspected bots rule to prevent any access from bots that are not on the good bots list.

    Hope this helps.

    Eldad

     

  • Avatar
    zeroone

    Under the Block Bad Bots list.  Can I enter bots or any information here even ones that are not known to Incapsula?  I want to block specific traffic from visiting my website. I know their user agent information and referrer. But there is no where in Incapsula where I can block specific User Agents Or referrers.  Can i put that under the bad bots lists and will this tool take whatever is in the string I enter and if Incapsula sees any of the matching string in the header it will reject access to my website?

  • Avatar
    Arkady

    You can contact Incapsula Support via support@incapsula.com and ask to define the specific user agent.

    Arkady  

  • Avatar
    Aghassiv

    Sometimes Incapsula blocks even humans or bots which I would like to allow. I can't reverse the feature of blocking bad bots to “Observe and Report” as described in the Guide.

     

  • Avatar
    Tantrumedia

    I am trying to verify a site using webmaster tools. It seems Incapsula is blocking whichever bot Google uses to identify ownership. I've tried adding a specific html file to the domain, and adding a meta-tag to the index.php page. Neither work, as I suspect Incapsula is not allowing the bot through. Surely Webmaster Tools is a 'good' bot?

  • Avatar
    nikolaiaas

    Is it possible to add complete folders to whitelists?
    So even bots can access a folder?

    Eg /allaccess/*

  • Avatar
    Mark

    If geographic restrictions are enabled, say blocking traffic from the US, can I still allow Good Bots through from the blocked region?

  • Avatar
    Beny

    Hello Mark,
    All Good Bots (like Google and Pingdom) will be allowed to access your site:
    anyway .
    If you would like to disable one of them , you need to uncheck it.
    ====================detailed description ===================
    Bots listed here will not be blocked (Checked by default).
    “Good Bots” list consists of Bots that are considered beneficial
    If you wish to block specific Bots, click “Good Bots…” and uncheck a Bot that you would like to block from accessing your site.
    Users cannot disable this feature directly, but can manually remove all Bots from “Good Bots” list

  • Avatar
    Beny

    Hello Mark ,
    An update,
    While blocking traffic from the US, activated,and Good Bots, needed to be allowed .
    You need to add wanted bots as exceptions ,in the same section below.

  • Avatar
    Michael

    If Cyclone is a known DDOS bot, why is it required to add it in manually to the bad bot list??

  • Avatar
    Ronen - Support

    Hello Michael,

    By definition, the impact of DDoS bots is significant when a DDoS attack is in progress.
    Once a DDoS attack is detected by Incapsula, bots which are classified as DDoS bots are immediately blocked in order to avoid further impact on the targeted site.
    Incapsula provides customers the option of blocking these bots completely, also in cases where a DDoS attack is not occurring.
    Hope this answers you question.
    Feel free to ask any more questions.

Powered by Zendesk