Package com.norconex.crawler.web.fetch.impl.webdriver

Class WebDriverFetcherConfig

java.lang.Object
com.norconex.crawler.core.fetch.BaseFetcherConfig
com.norconex.crawler.web.fetch.impl.webdriver.WebDriverFetcherConfig
public class WebDriverFetcherConfig extends BaseFetcherConfig

Configuration for WebDriverFetcher.

Since:
3.0.0
See Also:

  • Constructor Details

    • WebDriverFetcherConfig

      public WebDriverFetcherConfig()

  • Method Details

    • getCapabilities

      public Map<String,String> getCapabilities()
      Gets optional capabilities (configuration options) for the web driver. Many are specific to each browser or web driver. Refer to vendor documentation.
      Returns:
      capabilities

    • setCapabilities

      public WebDriverFetcherConfig setCapabilities(Map<String,String> capabilities)
      Sets optional capabilities (configuration options) for the web driver. Many are specific to each browser or web driver. Refer to vendor documentation.
      Parameters:
      capabilities - web driver capabilities
      Returns:
      this

    • getBrowser

      public WebDriverBrowser getBrowser()
      The browser used for crawling. Also defines which WebDriver to use. Default is Firefox.

    • getDriverPath

      public Path getDriverPath()
      Local path to driver executable or null to attempt automatic detection of the driver path. See web driver vendor documentation for the location facilitating detection. Use setRemoteURL(URL) instead when using a remote web driver cluster.

    • getBrowserPath

      public Path getBrowserPath()
      Local path to browser executable or null to attempt automatic browser path detection. See browser vendor documentation for the expected browser installed location. Use setRemoteURL(URL) instead when using a remote web driver cluster.

    • getRemoteURL

      public URL getRemoteURL()
      URL of a remote WebDriver cluster. Alternative to using a local browser and local web driver.

    • isUseHtmlUnit

      public boolean isUseHtmlUnit()
      Experimental feature where the selected WebDriverBrowser will be used to configure and use HtmlUnit WebDriver as a wrapper instead of directly using the browser WebDriver. Not all browsers are supported by HtmlUnit (Chrome, Firefox, and Edge are, as of this writing).

    • getHttpSniffer

      public HttpSniffer getHttpSniffer()
      Optionally setup an HTTP proxy that allows to set and capture HTTP headers. For advanced use only.

    • getScreenshotHandler

      public WebDriverScreenshotHandler getScreenshotHandler()
      When configured, takes screenshots of each web pages.

    • getArguments

      public List<String> getArguments()
      Optional command-line arguments supported by some web driver or browser.

    • getWindowSize

      public Dimension getWindowSize()
      Optionally set the browser window dimensions. E.g., 640x480.

    • getEarlyPageScript

      public String getEarlyPageScript()
      Optional JavaScript code to be run the moment a page is requested.

    • getLatePageScript

      public String getLatePageScript()
      Optional JavaScript code to be run after we are done waiting for a page.

    • getPageLoadTimeout

      public Duration getPageLoadTimeout()
      Web driver max wait time for a page to load.

    • getImplicitlyWait

      public Duration getImplicitlyWait()
      Web driver max wait time for an element to appear. See getWaitForElementSelector().

    • getScriptTimeout

      public Duration getScriptTimeout()
      Web driver max wait time for a scripts to execute.

    • getThreadWait

      public Duration getThreadWait()
      Makes the current thread sleep for the specified duration, to give the web driver enough time to load the page. Sometimes necessary for some web driver implementations when preferable options fail.

    • getWaitForElementType

      public WebDriverFetcherConfig.WaitElementType getWaitForElementType()
      The type of reference to use when waiting for an element.

    • getWaitForElementSelector

      public String getWaitForElementSelector()
      Reference to an element to wait for. The nature of the reference itself is defined by getWaitForElementType().

    • getWaitForElementTimeout

      public Duration getWaitForElementTimeout()
      Max wait time for an element to show up in browser before returning. Default 'type' is 'tagName'.

    • getBrowserMaxNavigations

      public int getBrowserMaxNavigations()
      A maximum number of navigations at which we restart the browser. Applies to individual browser instances, typically one per crawl threads. After a while, browsers can accumulate a lot of data in memory and may start performing poorly or cause other issues. Restarting may help. Default is -1 for unlimited navigation (no restart)

    • getBrowserMaxAge

      public Duration getBrowserMaxAge()
      A maximum duration at which we restart the browser. Applies to individual browser instances, typically one per crawl threads. After a while, browsers can accumulate a lot of data in memory and may start performing poorly or cause other issues. Restarting may help. Default is null (no restart)

    • getCleanupInterval

      @NonNull public @NonNull Duration getCleanupInterval()
      Amount of time between each check for unused browser instances and closing them. Defaults to 10 seconds. Cannot be null.

    • setBrowser

      public WebDriverFetcherConfig setBrowser(WebDriverBrowser browser)
      The browser used for crawling. Also defines which WebDriver to use. Default is Firefox.
      Returns:
      this.

    • setDriverPath

      public WebDriverFetcherConfig setDriverPath(Path driverPath)
      Local path to driver executable or null to attempt automatic detection of the driver path. See web driver vendor documentation for the location facilitating detection. Use setRemoteURL(URL) instead when using a remote web driver cluster.
      Returns:
      this.

    • setBrowserPath

      public WebDriverFetcherConfig setBrowserPath(Path browserPath)
      Local path to browser executable or null to attempt automatic browser path detection. See browser vendor documentation for the expected browser installed location. Use setRemoteURL(URL) instead when using a remote web driver cluster.
      Returns:
      this.

    • setRemoteURL

      public WebDriverFetcherConfig setRemoteURL(URL remoteURL)
      URL of a remote WebDriver cluster. Alternative to using a local browser and local web driver.
      Returns:
      this.

    • setUseHtmlUnit

      public WebDriverFetcherConfig setUseHtmlUnit(boolean useHtmlUnit)
      Experimental feature where the selected WebDriverBrowser will be used to configure and use HtmlUnit WebDriver as a wrapper instead of directly using the browser WebDriver. Not all browsers are supported by HtmlUnit (Chrome, Firefox, and Edge are, as of this writing).
      Returns:
      this.

    • setHttpSniffer

      public WebDriverFetcherConfig setHttpSniffer(HttpSniffer httpSniffer)
      Optionally setup an HTTP proxy that allows to set and capture HTTP headers. For advanced use only.
      Returns:
      this.

    • setScreenshotHandler

      public WebDriverFetcherConfig setScreenshotHandler(WebDriverScreenshotHandler screenshotHandler)
      When configured, takes screenshots of each web pages.
      Returns:
      this.

    • setWindowSize

      public WebDriverFetcherConfig setWindowSize(Dimension windowSize)
      Optionally set the browser window dimensions. E.g., 640x480.
      Returns:
      this.

    • setEarlyPageScript

      public WebDriverFetcherConfig setEarlyPageScript(String earlyPageScript)
      Optional JavaScript code to be run the moment a page is requested.
      Returns:
      this.

    • setLatePageScript

      public WebDriverFetcherConfig setLatePageScript(String latePageScript)
      Optional JavaScript code to be run after we are done waiting for a page.
      Returns:
      this.

    • setPageLoadTimeout

      public WebDriverFetcherConfig setPageLoadTimeout(Duration pageLoadTimeout)
      Web driver max wait time for a page to load.
      Returns:
      this.

    • setImplicitlyWait

      public WebDriverFetcherConfig setImplicitlyWait(Duration implicitlyWait)
      Web driver max wait time for an element to appear. See getWaitForElementSelector().
      Returns:
      this.

    • setScriptTimeout

      public WebDriverFetcherConfig setScriptTimeout(Duration scriptTimeout)
      Web driver max wait time for a scripts to execute.
      Returns:
      this.

    • setThreadWait

      public WebDriverFetcherConfig setThreadWait(Duration threadWait)
      Makes the current thread sleep for the specified duration, to give the web driver enough time to load the page. Sometimes necessary for some web driver implementations when preferable options fail.
      Returns:
      this.

    • setWaitForElementType

      public WebDriverFetcherConfig setWaitForElementType(WebDriverFetcherConfig.WaitElementType waitForElementType)
      The type of reference to use when waiting for an element.
      Returns:
      this.

    • setWaitForElementSelector

      public WebDriverFetcherConfig setWaitForElementSelector(String waitForElementSelector)
      Reference to an element to wait for. The nature of the reference itself is defined by getWaitForElementType().
      Returns:
      this.

    • setWaitForElementTimeout

      public WebDriverFetcherConfig setWaitForElementTimeout(Duration waitForElementTimeout)
      Max wait time for an element to show up in browser before returning. Default 'type' is 'tagName'.
      Returns:
      this.

    • setBrowserMaxNavigations

      public WebDriverFetcherConfig setBrowserMaxNavigations(int browserMaxNavigations)
      A maximum number of navigations at which we restart the browser. Applies to individual browser instances, typically one per crawl threads. After a while, browsers can accumulate a lot of data in memory and may start performing poorly or cause other issues. Restarting may help. Default is -1 for unlimited navigation (no restart)
      Returns:
      this.

    • setBrowserMaxAge

      public WebDriverFetcherConfig setBrowserMaxAge(Duration browserMaxAge)
      A maximum duration at which we restart the browser. Applies to individual browser instances, typically one per crawl threads. After a while, browsers can accumulate a lot of data in memory and may start performing poorly or cause other issues. Restarting may help. Default is null (no restart)
      Returns:
      this.

    • setCleanupInterval

      public WebDriverFetcherConfig setCleanupInterval(@NonNull @NonNull Duration cleanupInterval)
      Amount of time between each check for unused browser instances and closing them. Defaults to 10 seconds. Cannot be null.
      Returns:
      this.

    • equals

      public boolean equals(Object o)
      Overrides:
      equals in class BaseFetcherConfig

    • canEqual

      protected boolean canEqual(Object other)
      Overrides:
      canEqual in class BaseFetcherConfig

    • hashCode

      public int hashCode()
      Overrides:
      hashCode in class BaseFetcherConfig

    • toString

      public String toString()
      Overrides:
      toString in class BaseFetcherConfig