WebDriverFetcher

Fetches dynamically generated web pages using a WebDriver-based browser.

crawler/webJavadoc: WebDriverFetcher

Notes

Fetches dynamically generated web pages using a WebDriver-based browser (such as Firefox or Chrome via Selenium). Supports executing custom JavaScript before and after page load, waiting for specific DOM elements, HTTP traffic sniffing, screenshot capture, and remote WebDriver configuration.

Examples

YAML

class: WebDriverFetcher
browser: FIREFOX
driverPath: /drivers/geckodriver.exe
referenceFilters:
  - class: GenericReferenceFilter
    valueMatcher:
      method: REGEX
      pattern: .*dynamic.*$

JSON

{
  "class": "WebDriverFetcher",
  "browser": "FIREFOX",
  "driverPath": "/drivers/geckodriver.exe",
  "referenceFilters": [
    {
      "class": "GenericReferenceFilter",
      "valueMatcher": {
        "method": "REGEX",
        "pattern": ".*dynamic.*$"
      }
    }
  ]
}

XML

<class>WebDriverFetcher</class>
<browser>FIREFOX</browser>
<driverPath>/drivers/geckodriver.exe</driverPath>
<referenceFilters>
  <referenceFilter>
    <class>GenericReferenceFilter</class>
    <valueMatcher>
      <method>REGEX</method>
      <pattern>.*dynamic.*$</pattern>
    </valueMatcher>
  </referenceFilter>
</referenceFilters>

Usage

Full configuration skeleton, with every option and its default

YAML

class: WebDriverFetcher
arguments:
  - string
browser: CHROME
browserMaxAge: 0
browserMaxNavigations: 0
browserPath: string
capabilities: {}
cleanupInterval: 0
driverPath: string
earlyPageScript: string
httpSniffer:
  chainedProxy:
    credentials: {}
    host: {}
    realm: string
    scheme: string
  host: string
  maxBufferSize: 0
  networkInterface: string
  port: 0
  requestHeaders: {}
  responseTimeout: 0
  userAgent: string
implicitlyWait: 0
latePageScript: string
pageLoadTimeout: 0
referenceFilters: []
remoteURL: string
screenshotHandler:
  cssSelector: string
  imageFormat: string
  targetDir: string
  targetDirField: string
  targetDirStructure: URL2PATH
  targetMetaField: string
  targets:
    - METADATA
scriptTimeout: 0
threadWait: 0
useHtmlUnit: false
waitForElementSelector: string
waitForElementTimeout: 0
waitForElementType: TAGNAME
windowSize: string

JSON

{
  "class": "WebDriverFetcher",
  "arguments": [
    "string"
  ],
  "browser": "CHROME",
  "browserMaxAge": 0,
  "browserMaxNavigations": 0,
  "browserPath": "string",
  "capabilities": {},
  "cleanupInterval": 0,
  "driverPath": "string",
  "earlyPageScript": "string",
  "httpSniffer": {
    "chainedProxy": {
      "credentials": {},
      "host": {},
      "realm": "string",
      "scheme": "string"
    },
    "host": "string",
    "maxBufferSize": 0,
    "networkInterface": "string",
    "port": 0,
    "requestHeaders": {},
    "responseTimeout": 0,
    "userAgent": "string"
  },
  "implicitlyWait": 0,
  "latePageScript": "string",
  "pageLoadTimeout": 0,
  "referenceFilters": [],
  "remoteURL": "string",
  "screenshotHandler": {
    "cssSelector": "string",
    "imageFormat": "string",
    "targetDir": "string",
    "targetDirField": "string",
    "targetDirStructure": "URL2PATH",
    "targetMetaField": "string",
    "targets": [
      "METADATA"
    ]
  },
  "scriptTimeout": 0,
  "threadWait": 0,
  "useHtmlUnit": false,
  "waitForElementSelector": "string",
  "waitForElementTimeout": 0,
  "waitForElementType": "TAGNAME",
  "windowSize": "string"
}

XML

<webDriverFetcher>
  <class>WebDriverFetcher</class>
  <arguments>
    <argument>string</argument>
  </arguments>
  <browser>CHROME</browser>
  <browserMaxAge>0</browserMaxAge>
  <browserMaxNavigations>0</browserMaxNavigations>
  <browserPath>string</browserPath>
  <capabilities/>
  <cleanupInterval>0</cleanupInterval>
  <driverPath>string</driverPath>
  <earlyPageScript>string</earlyPageScript>
  <httpSniffer>
    <chainedProxy>
      <credentials/>
      <host/>
      <realm>string</realm>
      <scheme>string</scheme>
    </chainedProxy>
    <host>string</host>
    <maxBufferSize>0</maxBufferSize>
    <networkInterface>string</networkInterface>
    <port>0</port>
    <requestHeaders/>
    <responseTimeout>0</responseTimeout>
    <userAgent>string</userAgent>
  </httpSniffer>
  <implicitlyWait>0</implicitlyWait>
  <latePageScript>string</latePageScript>
  <pageLoadTimeout>0</pageLoadTimeout>
  <referenceFilters>
    <referenceFilter/>
  </referenceFilters>
  <remoteURL>string</remoteURL>
  <screenshotHandler>
    <cssSelector>string</cssSelector>
    <imageFormat>string</imageFormat>
    <targetDir>string</targetDir>
    <targetDirField>string</targetDirField>
    <targetDirStructure>URL2PATH</targetDirStructure>
    <targetMetaField>string</targetMetaField>
    <targets>
      <target>METADATA</target>
    </targets>
  </screenshotHandler>
  <scriptTimeout>0</scriptTimeout>
  <threadWait>0</threadWait>
  <useHtmlUnit>false</useHtmlUnit>
  <waitForElementSelector>string</waitForElementSelector>
  <waitForElementTimeout>0</waitForElementTimeout>
  <waitForElementType>TAGNAME</waitForElementType>
  <windowSize>string</windowSize>
</webDriverFetcher>

Properties

Property	Type	Required	Default
arguments	string[]	No	-
browser	enum	No	FIREFOX
browserMaxAge	integer	No	-
browserMaxNavigations	integer	No	-1
browserPath	string	No	-
capabilities	Record<string, string>	No	-
cleanupInterval	integer	No	10000
driverPath	string	No	-
earlyPageScript	string	No	-
httpSniffer	HttpSniffer	No	-
implicitlyWait	integer	No	-
latePageScript	string	No	-
pageLoadTimeout	integer	No	-
referenceFilters	ReferenceFilter[]	No	-
remoteURL	string	No	-
screenshotHandler	WebDriverScreenshotHandler	No	-
scriptTimeout	integer	No	-
threadWait	integer	No	-
useHtmlUnit	boolean	No	false
waitForElementSelector	string	No	-
waitForElementTimeout	integer	No	-
waitForElementType	enum	No	-
windowSize	string	No	-

Property Details

arguments

Optional command-line arguments passed to the browser or web driver executable (e.g., --headless, --no-sandbox).

browser

The browser to use for crawling, which also determines the WebDriver to use. Default is FIREFOX.

Allowed Values

• CHROME
• FIREFOX
• EDGE
• SAFARI
• OPERA
• CUSTOM

browserMaxAge

The maximum duration a browser instance is kept alive before being restarted. Restarting browsers periodically helps prevent memory accumulation and performance degradation. Default is null (no age-based restart).

browserMaxNavigations

The maximum number of page navigations before a browser instance is restarted. Helps prevent memory issues with long-running browsers. Default is -1 (unlimited, no restart).

browserPath

The local file path to the browser executable. When not set, the browser is expected to be found automatically. Not used when remoteURL is configured.

capabilities

Optional key-value pairs of browser or driver capabilities (configuration options). Many are browser-specific; refer to vendor documentation for supported values.

cleanupInterval

The interval between checks for idle browser instances that can be closed and cleaned up. Default is 10 seconds.

driverPath

The local file path to the WebDriver executable (e.g., geckodriver, chromedriver). When not set, automatic detection is attempted. Not used when remoteURL is configured.

earlyPageScript

Optional JavaScript code executed immediately after a page is requested, before waiting for the page to fully load.

httpSniffer

An optional HTTP traffic sniffer configured to intercept browser requests and responses, enabling capture of HTTP headers and other traffic details.

implicitlyWait

The WebDriver implicit wait timeout — how long the driver will wait for an element to appear when searching before throwing an exception.

latePageScript

Optional JavaScript code executed after the page load wait completes, just before the page content is captured.

pageLoadTimeout

The maximum time the WebDriver will wait for a page to fully load before proceeding.

referenceFilters

Filters applied to document references (URLs or file paths) to control which ones are fetched. Documents whose references do not pass the filters are skipped.

remoteURL

The URL of a remote WebDriver cluster (e.g., Selenium Grid). When set, the fetcher connects to this remote instance instead of using a local browser and driver.

screenshotHandler

When configured, captures a screenshot of each crawled web page. Specify a WebDriverScreenshotHandler with target directory and format settings.

scriptTimeout

The maximum time the WebDriver will wait for a JavaScript script to finish executing.

threadWait

A simple thread sleep duration to give the browser extra time to load the page. Use as a last resort when other wait mechanisms are insufficient for a particular browser or web driver.

useHtmlUnit

When enabled, uses HtmlUnit WebDriver as a wrapper around the selected browser instead of direct browser WebDriver. This is an experimental feature and not all browsers are supported.

waitForElementSelector

A selector expression used to identify an element that must appear in the browser before proceeding. The type of selector is determined by waitForElementType.

waitForElementTimeout

The maximum time to wait for the element specified by waitForElementSelector to appear in the browser before timing out.

waitForElementType

The type of reference used by waitForElementSelector to locate the element to wait for. Supported values include TAGNAME, CLASSNAME, CSSSELECTOR, ID, LINKTEXT, NAME, PARTIALLINKTEXT, and XPATH. Default is TAGNAME.

Allowed Values

• TAGNAME
• CLASSNAME
• CSSSELECTOR
• ID
• LINKTEXT
• NAME
• PARTIALLINKTEXT
• XPATH

windowSize

The browser window dimensions (width x height in pixels) to use when rendering pages. Useful for controlling responsive layout behavior.