Skip to main content

HttpClientFetcher

Fetches web documents using an HTTP client with authentication and SSL support.

crawler/webJavadoc: HttpClientFetcher

Notes​

Fetches web documents using an HTTP client. Supports authentication (BASIC, DIGEST, NTLM, form-based, Kerberos), SSL configuration, cookie handling, ETag and If-Modified-Since headers, proxy settings, redirect handling, connection pooling, and custom request headers.

Examples​

class: HttpClientFetcher
authentication:
  method: FORM
  credentials:
    username: joeUser
    password: joePassword
  formUsernameField: loginUser
  formPasswordField: loginPwd
  url: http://www.example.com/login/submit

Usage​

Full configuration skeleton, with every option and its default
class: HttpClientFetcher
authentication:
  credentials:
    password: string
    passwordKey: {}
    username: string
  domain: string
  formCharset: string
  formParams: {}
  formPasswordField: string
  formSelector: string
  formUsernameField: string
  host:
    name: string
    port: 0
  kerberosConfig:
    keytabPath: string
    krb5ConfigPath: string
    loginModuleName: string
    principal: string
    servicePrincipalName: string
    useTicketCache: false
  method: FORM
  preemptive: false
  realm: string
  url: string
  workstation: string
connectionRequestTimeout: 0
connectionTimeout: 0
cookieSpec: RELAXED
etagDisabled: false
expectContinueEnabled: false
forceCharsetDetection: false
forceContentTypeDetection: false
headersPrefix: string
hstsDisabled: false
httpMethods:
  - GET
ifModifiedSinceDisabled: false
localAddress: string
maxConnectionIdleTime: 0
maxConnectionInactiveTime: 0
maxConnections: 0
maxConnectionsPerRoute: 0
maxRedirects: 0
notFoundStatusCodes:
  - 0
proxySettings:
  credentials:
    password: string
    passwordKey: {}
    username: string
  host:
    name: string
    port: 0
  realm: string
  scheme: string
redirectUrlProvider:
  class: GenericRedirectUrlProvider
  fallbackCharset: string
referenceFilters: []
requestHeaders: {}
sniDisabled: false
socketTimeout: 0
sslProtocols:
  - string
trustAllSSLCertificates: false
userAgent: string
validStatusCodes:
  - 0

Properties​

PropertyTypeRequiredDefault
authenticationHttpAuthConfigNo-
connectionRequestTimeoutintegerNo30000
connectionTimeoutintegerNo30000
cookieSpecenumNoRELAXED
etagDisabledbooleanNo-
expectContinueEnabledbooleanNofalse
forceCharsetDetectionbooleanNofalse
forceContentTypeDetectionbooleanNofalse
headersPrefixstringNo-
hstsDisabledbooleanNofalse
httpMethodsenum[]No[GET, HEAD]
ifModifiedSinceDisabledbooleanNofalse
localAddressstringNo-
maxConnectionIdleTimeintegerNo10000
maxConnectionInactiveTimeintegerNo-
maxConnectionsintegerNo200
maxConnectionsPerRouteintegerNo20
maxRedirectsintegerNo50
notFoundStatusCodesinteger[]No[404]
proxySettingsProxySettingsNoProxySettings
redirectUrlProviderRedirectUrlProviderNoGenericRedirectUrlProvider
referenceFiltersReferenceFilter[]No-
requestHeadersRecord<string, string>No-
sniDisabledbooleanNofalse
socketTimeoutintegerNo30000
sslProtocolsstring[]No-
trustAllSSLCertificatesbooleanNofalse
userAgentstringNo-
validStatusCodesinteger[]No[200]

Property Details​

authentication​

Authentication configuration for sites requiring login. Supports methods such as BASIC, DIGEST, NTLM, form-based, and Kerberos. Default is null (no authentication).

connectionRequestTimeout​

The maximum time to wait when requesting a connection from the connection pool. Default is 30 seconds.

connectionTimeout​

The maximum time to wait for a connection to be established with the target server. Default is 30 seconds.

cookieSpec​

The cookie specification policy to use when handling cookies. Supported values are RELAXED (default, lenient parsing), STRICT (strict RFC-compliant parsing), and IGNORE (cookies are ignored entirely).

Allowed Values​

  • RELAXED
  • STRICT
  • IGNORE

etagDisabled​

When enabled, the If-None-Match ETag HTTP request header is not sent. By default, ETags are used so servers can skip sending unchanged content.

expectContinueEnabled​

When enabled, the Expect: 100-continue handshake is used before sending the request body, allowing the server to reject the request early based on headers alone.

forceCharsetDetection​

When enabled, the character encoding is detected from content rather than relying on the charset value in the Content-Type HTTP response header.

forceContentTypeDetection​

When enabled, the content type is detected from the document content rather than relying on the Content-Type HTTP response header.

headersPrefix​

An optional prefix to prepend to HTTP response header field names when storing them as document metadata. Default is null (no prefix added).

hstsDisabled​

When enabled, HTTP Strict Transport Security (HSTS) enforcement is disabled, preventing the crawler from automatically upgrading HTTP URLs to HTTPS based on the Strict-Transport-Security response header.

httpMethods​

The list of HTTP methods accepted by this fetcher. Defaults to GET and HEAD.

ifModifiedSinceDisabled​

When enabled, the If-Modified-Since HTTP request header is not sent. By default, this header is sent so servers can respond with 304 Not Modified when content has not changed.

localAddress​

The local network address (IP) to use for outbound connections. Useful when the machine has multiple network interfaces and a specific one should be used.

maxConnectionIdleTime​

The period of time after which idle connections are evicted from the connection pool. Default is 10 seconds.

maxConnectionInactiveTime​

The period of inactivity after which a connection is proactively checked to detect stale connections. Default is 0 (not proactively checked).

maxConnections​

The maximum total number of connections that can be created in the connection pool. Typically set to at least the number of crawler threads. Default is 200.

maxConnectionsPerRoute​

The maximum number of concurrent connections to any single host/route. Default is 20.

maxRedirects​

The maximum number of redirects to follow before giving up, which helps prevent infinite redirect loops. Set to zero to disable redirect following. Default is 50.

notFoundStatusCodes​

HTTP response status codes to be treated as "not found", causing the document to be marked as deleted. Default is [404].

proxySettings​

Configures an optional HTTP proxy for outbound connections. Leave unset to connect directly.

redirectUrlProvider​

Provides the target URL to use when a redirect is encountered. Defaults to GenericRedirectUrlProvider.

referenceFilters​

Filters applied to document references (URLs or file paths) to control which ones are fetched. Documents whose references do not pass the filters are skipped.

requestHeaders​

Custom HTTP headers to be sent with every HTTP request, in addition to default headers provided by the HTTP client.

sniDisabled​

When enabled, Server Name Indication (SNI) is disabled for SSL/TLS connections. SNI is required for some HTTPS sites that host multiple domains on a single IP.

socketTimeout​

The maximum time in milliseconds to wait for data on an established connection before giving up.

sslProtocols​

The list of SSL/TLS protocols to support (e.g., TLSv1.2, TLSv1.3). Default is null, which uses the protocols configured by the Java platform.

trustAllSSLCertificates​

When enabled, all SSL certificates are trusted, including self-signed or invalid ones. This is a security risk and should only be used in controlled environments. Consider installing a proper certificate instead.

userAgent​

The user-agent string used to identify the crawler to target web sites. It is highly recommended to always set a meaningful user agent so web site administrators can identify the crawler.

validStatusCodes​

HTTP response status codes considered successful. Documents with other status codes are not processed normally. Default is [200].