Skip to main content

PlaywrightFetcher

Uses Microsoft Playwright for browser-based crawling, driving real browser engines (Chromium, Firefox, WebKit) across Windows, macOS, and Linux. Unlike WebDriverFetcher, no exte...

crawler/webJavadoc: PlaywrightFetcher

Notes​

By default, Playwright downloads and uses its own browser binaries stored under ~/.cache/ms-playwright. To use a system-installed browser instead (e.g., Chrome, Edge, or Firefox already present on the machine or CI runner), set the channel property (e.g., chrome, msedge, firefox).

Because Playwright manages one browser instance per thread, it is safe to use with multi-threaded crawling. Browsers are automatically restarted after browserMaxNavigations navigations or after they exceed browserMaxAge, whichever comes first.

Examples​

Chromium with element wait​

class: PlaywrightFetcher
browser: CHROMIUM
headless: true
waitForElementType: CSSSELECTOR
waitForElementSelector: "#main-content"
waitForElementTimeout: PT10S
referenceFilters:
- class: GenericReferenceFilter
  valueMatcher:
    method: REGEX
    pattern: ".*dynamic.*"

Usage​

Full configuration skeleton, with every option and its default
class: PlaywrightFetcher
args:
  - string
browser: CHROMIUM
browserMaxAge: 0
browserMaxNavigations: 0
channel: string
cleanupInterval: 0
earlyPageScript: string
executablePath: string
headless: false
ignoreHttpsErrors: false
latePageScript: string
pageLoadTimeout: 0
proxySettings:
  credentials:
    password: string
    passwordKey: {}
    username: string
  host:
    name: string
    port: 0
  realm: string
  scheme: string
referenceFilters: []
screenshotHandler:
  configuration:
    cssSelector: string
    imageFormat: string
    targetDir: string
    targetDirField: string
    targetDirStructure: URL2PATH
    targetMetaField: string
    targets: []
slowMo: 0
waitForElementSelector: string
waitForElementTimeout: 0
waitForElementType: TAGNAME
windowSize: string

Properties​

PropertyTypeRequiredDefault
argsstring[]No-
browserenumNoCHROMIUM
browserMaxAgeintegerNo-
browserMaxNavigationsintegerNo-1
channelstringNo-
cleanupIntervalintegerNo10000
earlyPageScriptstringNo-
executablePathstringNo-
headlessbooleanNotrue
ignoreHttpsErrorsbooleanNofalse
latePageScriptstringNo-
pageLoadTimeoutintegerNo-
proxySettingsProxySettingsNo-
referenceFiltersReferenceFilter[]No-
screenshotHandlerPlaywrightScreenshotHandlerNo-
slowMointegerNo-
waitForElementSelectorstringNo-
waitForElementTimeoutintegerNo-
waitForElementTypeenumNoCSSSELECTOR
windowSizestringNo-

Property Details​

args​

Optional command-line arguments passed to the browser at launch (e.g., --disable-extensions).

browser​

The browser engine to use for crawling. Supported values are CHROMIUM (default), FIREFOX, and WEBKIT. Ignored when channel is set.

Allowed Values​

  • CHROMIUM
  • FIREFOX
  • WEBKIT

browserMaxAge​

Maximum age of a per-thread browser instance before it is automatically restarted. Helps prevent memory accumulation over long crawls. Default is no age-based restart.

browserMaxNavigations​

Maximum number of page navigations per thread browser instance before it is automatically restarted. Helps prevent memory accumulation. Default is -1 (unlimited).

channel​

Optional browser channel that instructs Playwright to use a system-installed browser instead of its own downloaded binaries. Supported values: chrome, msedge, firefox. Leave blank (the default) to use Playwright's bundled browser binaries.

cleanupInterval​

How often idle (unused) per-thread browser instances are checked and closed. Defaults to 10 seconds.

earlyPageScript​

Optional JavaScript snippet executed immediately after page navigation, before any wait conditions (such as waitForElementSelector) are applied. Useful for dismissing popups or injecting page-level state.

executablePath​

Optional path to a custom browser executable. When not set, Playwright uses the bundled browser binary it downloaded.

headless​

Whether to run the browser in headless (no visible window) mode. Default is true.

ignoreHttpsErrors​

When true, HTTPS certificate errors (e.g., self-signed or expired certificates) are ignored. Default is false.

latePageScript​

Optional JavaScript snippet executed after all wait conditions have been satisfied. Useful for final page manipulation before the content is captured.

pageLoadTimeout​

Maximum time to wait for the page load event to fire. When not set, the Playwright default (30 seconds) applies.

proxySettings​

Proxy settings used when launching the browser. When not configured, no proxy is applied.

referenceFilters​

One or more reference filters that restrict which URLs this fetcher will handle. When multiple fetchers are configured, each document URL is offered to fetchers in order; the first fetcher whose filters accept the URL handles the request.

screenshotHandler​

When configured, captures a screenshot of each fetched page and stores it according to the handler's settings (document metadata field or local directory).

slowMo​

Optional slow-motion delay applied between Playwright operations. Useful for debugging or visually tracing a crawl. Default is no delay.

waitForElementSelector​

Selector expression used to wait for a specific element to appear in the page before the content is captured. The type of expression is controlled by waitForElementType. When not set, no element wait is performed.

waitForElementTimeout​

Maximum time to wait for the element identified by waitForElementSelector to appear. When not set, the Playwright default timeout applies.

waitForElementType​

The locator strategy used by waitForElementSelector. Supported values: TAGNAME, CLASSNAME, CSSSELECTOR (default), ID, LINKTEXT, NAME, PARTIALLINKTEXT, XPATH.

Allowed Values​

  • TAGNAME
  • CLASSNAME
  • CSSSELECTOR
  • ID
  • LINKTEXT
  • NAME
  • PARTIALLINKTEXT
  • XPATH

windowSize​

Optional browser viewport size. When not set, the Playwright default viewport is used.