Answer a question

Is there a way to make your Selenium script undetectable in Python using geckodriver?

I'm using Selenium for scraping. Are there any protections we need to use so websites can't detect Selenium?

Answers

The fact that selenium driven Firefox / GeckoDriver gets detected doesn't depends on any specific GeckoDriver or Firefox version. The Websites themselves can detect the network traffic and can identify the Browser Client i.e. Web Browser as WebDriver controled.

As per the documentation of the WebDriver Interface in the latest editor's draft of WebDriver - W3C Living Document the webdriver-active flag which is initially set as false, is set to true when the user agent is under remote control i.e. when controlled through Selenium.

NavigatorAutomationInformation

Now that the NavigatorAutomationInformation interface should not be exposed on WorkerNavigator.

mixin NavigatorAutomationInformation

So,

webdriver
    Returns true if webdriver-active flag is set, false otherwise.

where as,

navigator.webdriver
    Defines a standard way for co-operating user agents to inform the document that it is controlled by WebDriver, for example so that alternate code paths can be triggered during automation.

So, the bottom line is:

Selenium identifies itself


However some generic approaches to avoid getting detected while web-scraping are as follows:

  • The first and foremost attribute a website can determine your script/program is through your monitor size. So it is recommended not to use the conventional Viewport.
  • If you need to send multiple requests to a website, you need to keep on changing the User Agent on each request. Here you can find a detailed discussion on Way to change Google Chrome user agent in Selenium?
  • To simulate human like behavior you may require to slow down the script execution even beyond WebDriverWait and expected_conditions inducing time.sleep(secs). Here you can find a detailed discussion on How to sleep webdriver in python for milliseconds
Logo

学AI,认准AI Studio!GPU算力,限时免费领,邀请好友解锁更多惊喜福利 >>>

更多推荐