Package com.norconex.crawler.web.fetch.impl.webdriver

Class WebDriverFetcher

java.lang.Object
com.norconex.crawler.core.fetch.AbstractFetcher<WebDriverFetcherConfig>
com.norconex.crawler.web.fetch.impl.webdriver.WebDriverFetcher
All Implemented Interfaces:
Configurable<WebDriverFetcherConfig>, EventListener<Event>, Fetcher, EventListener, Consumer<Event>
public class WebDriverFetcher extends AbstractFetcher<WebDriverFetcherConfig>

Uses Selenium WebDriver support for using native browsers to crawl documents. Useful for crawling JavaScript-driven websites.

Considerations

Relying on an external software to fetch pages can be slower and not as scalable and may be less stable. Downloading of binaries and non-HTML file format may not always be possible. The use of HttpClientFetcher should be preferred whenever possible. Use at your own risk.

If it is important for each requests to be sharing the exact same browser session, consider reducing the number of threads to 1 since each thread has their own browser instance.

If you are using a Selenium grid and want to use multiple threads, make sure you configure Selenium max number of sessions per node to align with the number of threads you expect the crawler to use.

Supported HTTP method

This fetcher only supports HTTP GET method.

HTTP Headers

By default, web drivers do not expose HTTP headers from HTTP GET request. If you want to capture them, configure the "httpSniffer". A proxy service will be started to monitor HTTP traffic and store HTTP headers.

NOTE: Capturing headers with a proxy may not be supported by all Browsers/WebDriver implementations.

Since:
3.0.0