Package com.norconex.crawler.web.fetch.impl.httpclient

Class HttpClientFetcherConfig

java.lang.Object
com.norconex.crawler.core.fetch.BaseFetcherConfig
com.norconex.crawler.web.fetch.impl.httpclient.HttpClientFetcherConfig
public class HttpClientFetcherConfig extends BaseFetcherConfig
Generic HTTP Fetcher configuration.

  • Field Details

    • DEFAULT_TIMEOUT

      public static final Duration DEFAULT_TIMEOUT

    • DEFAULT_MAX_REDIRECT

      public static final int DEFAULT_MAX_REDIRECT
      See Also:

    • DEFAULT_MAX_CONNECTIONS

      public static final int DEFAULT_MAX_CONNECTIONS
      See Also:

    • DEFAULT_MAX_CONNECTIONS_PER_ROUTE

      public static final int DEFAULT_MAX_CONNECTIONS_PER_ROUTE
      See Also:

    • DEFAULT_MAX_IDLE_TIME

      public static final Duration DEFAULT_MAX_IDLE_TIME

    • DEFAULT_VALID_STATUS_CODES

      public static final List<Integer> DEFAULT_VALID_STATUS_CODES

    • DEFAULT_NOT_FOUND_STATUS_CODES

      public static final List<Integer> DEFAULT_NOT_FOUND_STATUS_CODES

  • Constructor Details

    • HttpClientFetcherConfig

      public HttpClientFetcherConfig()

  • Method Details

    • getValidStatusCodes

      public List<Integer> getValidStatusCodes()
      Sets valid HTTP response status codes.
      Returns:
      valid status codes

    • setValidStatusCodes

      public HttpClientFetcherConfig setValidStatusCodes(List<Integer> validStatusCodes)
      Gets valid HTTP response status codes.
      Parameters:
      validStatusCodes - valid status codes
      Returns:
      this

    • getNotFoundStatusCodes

      public List<Integer> getNotFoundStatusCodes()
      Gets HTTP status codes to be considered as "Not found" state. Default is 404.
      Returns:
      "Not found" codes

    • setNotFoundStatusCodes

      public final HttpClientFetcherConfig setNotFoundStatusCodes(List<Integer> notFoundStatusCodes)
      Sets HTTP status codes to be considered as "Not found" state.
      Parameters:
      notFoundStatusCodes - "Not found" codes
      Returns:
      this

    • setRequestHeader

      public HttpClientFetcherConfig setRequestHeader(String name, String value)
      Sets a default HTTP request header every HTTP connection should have. Those are in addition to any default request headers Apache HttpClient may already provide.
      Parameters:
      name - HTTP request header name
      value - HTTP request header value
      Returns:
      this

    • setRequestHeaders

      public HttpClientFetcherConfig setRequestHeaders(Map<String,String> headers)
      Sets a default HTTP request headers every HTTP connection should have. Those are in addition to any default request headers Apache HttpClient may already provide.
      Parameters:
      headers - map of header names and values
      Returns:
      this

    • getRequestHeader

      public String getRequestHeader(String name)
      Gets the HTTP request header value matching the given name, previously set with setRequestHeader(String, String).
      Parameters:
      name - HTTP request header name
      Returns:
      HTTP request header value or null if no match is found

    • getRequestHeaderNames

      public List<String> getRequestHeaderNames()
      Gets all HTTP request header names for headers previously set with setRequestHeader(String, String). If no request headers are set, it returns an empty array.
      Returns:
      HTTP request header names

    • removeRequestHeader

      public String removeRequestHeader(String name)
      Remove the request header matching the given name.
      Parameters:
      name - name of HTTP request header to remove
      Returns:
      the previous value associated with the name, or null if there was no request header for the name.

    • getProxySettings

      public ProxySettings getProxySettings()

    • setProxySettings

      public HttpClientFetcherConfig setProxySettings(ProxySettings proxy)

    • getSslProtocols

      public List<String> getSslProtocols()
      Gets the supported SSL/TLS protocols. Default is null, which means it will use those provided/configured by your Java platform.
      Returns:
      SSL/TLS protocols

    • setSslProtocols

      public HttpClientFetcherConfig setSslProtocols(List<String> sslProtocols)
      Sets the supported SSL/TLS protocols, such as SSLv3, TLSv1, TLSv1.1, and TLSv1.2. Note that specifying a protocol not supported by your underlying Java platform will not work.
      Parameters:
      sslProtocols - SSL/TLS protocols supported
      Returns:
      this

    • getHttpMethods

      public List<HttpMethod> getHttpMethods()
      Gets the list of HTTP methods to be accepted by this fetcher. Defaults are HttpMethod.GET and HttpMethod.HEAD.
      Returns:
      HTTP methods

    • setHttpMethods

      public HttpClientFetcherConfig setHttpMethods(List<HttpMethod> httpMethods)
      Sets the list of HTTP methods to be accepted by this fetcher. Defaults are HttpMethod.GET and HttpMethod.HEAD.
      Parameters:
      httpMethods - HTTP methods
      Returns:
      this

    • getHeadersPrefix

      public String getHeadersPrefix()
      Optional prefix prepended to captured HTTP response fields. A null value (default) won't add any prefix.

    • isForceContentTypeDetection

      public boolean isForceContentTypeDetection()
      Whether content type is detected instead of relying on returned Content-Type HTTP response header.

    • isForceCharsetDetection

      public boolean isForceCharsetDetection()
      Whether character encoding is detected instead of relying on the charset sometimes found in the Content-Type HTTP response header.

    • getAuthentication

      public HttpAuthConfig getAuthentication()
      Authentication configuration for sites requiring it. Default is null.

    • getCookieSpec

      public HttpClientFetcherConfig.CookieSpec getCookieSpec()
      Cookie specification to use when fetching documents. Default is relaxed.

    • getConnectionTimeout

      public Duration getConnectionTimeout()
      The connection timeout for a connection to be established. Default is DEFAULT_TIMEOUT.

    • getSocketTimeout

      public Duration getSocketTimeout()
      Gets the maximum period of inactivity between two consecutive data packets. Default is DEFAULT_TIMEOUT.

    • getConnectionRequestTimeout

      public Duration getConnectionRequestTimeout()
      Gets the timeout when requesting a connection. Default is DEFAULT_TIMEOUT.

    • getLocalAddress

      public String getLocalAddress()
      The local address, which may be useful when working with multiple network interfaces.

    • isExpectContinueEnabled

      public boolean isExpectContinueEnabled()
      Whether 'Expect: 100-continue' handshake is enabled. See RequestConfig.isExpectContinueEnabled()

    • getMaxRedirects

      public int getMaxRedirects()
      The maximum number of redirects to be followed. This can help prevent infinite loops. A value of zero effectively disables redirects. Default is DEFAULT_MAX_REDIRECT.

    • getMaxConnections

      public int getMaxConnections()
      The maximum number of connections that can be created. Typically, you would have at least the same amount as threads. Default is DEFAULT_MAX_CONNECTIONS.

    • getMaxConnectionsPerRoute

      public int getMaxConnectionsPerRoute()
      The maximum number of connections that can be used per route. Default is DEFAULT_MAX_CONNECTIONS_PER_ROUTE.

    • getMaxConnectionIdleTime

      public Duration getMaxConnectionIdleTime()
      Sets the period of time after which to evict idle connections from the connection pool. Default is DEFAULT_MAX_IDLE_TIME.

    • getMaxConnectionInactiveTime

      public Duration getMaxConnectionInactiveTime()
      Sets the period of time a connection must be inactive to be checked in case it became stalled. Default is 0 (not pro-actively checked).

    • getRequestHeaders

      public Map<String,String> getRequestHeaders()
      Headers to send with every HTTP request.

    • isIfModifiedSinceDisabled

      public boolean isIfModifiedSinceDisabled()
      Whether adding the If-Modified-Since HTTP request header is disabled. Servers supporting this header will only return the requested document if it was last modified since the supplied date.

    • isETagDisabled

      public boolean isETagDisabled()
      Whether adding "ETag" If-None-Match HTTP request header is disabled. Servers supporting this header will only return the requested document if the ETag value has changed, indicating a more recent version is available.

    • getUserAgent

      public String getUserAgent()
      The user-agent used when identifying the crawler to targeted web sites. It is highly recommended to always identify yourself.

    • getRedirectUrlProvider

      public RedirectUrlProvider getRedirectUrlProvider()
      The redirect URL provider. Defaults to GenericRedirectUrlProvider.

    • isTrustAllSSLCertificates

      public boolean isTrustAllSSLCertificates()
      Sets whether to trust all SSL certificate (affects only "https" connections). This is typically a bad idea if you care to avoid "man-in-the-middle" attacks. Try to install a SSL certificate locally to ensure a proper certificate exchange instead.
      Since:
      1.3.0

    • isSniDisabled

      public boolean isSniDisabled()
      Sets whether Server Name Indication (SNI) is disabled.

    • isHstsDisabled

      public boolean isHstsDisabled()
      Gets whether the forcing of non secure URLs to secure ones is disabled, according to the URL domain Strict-Transport-Security policy (obtained from HTTP response header).

    • setHeadersPrefix

      public HttpClientFetcherConfig setHeadersPrefix(String headersPrefix)
      Optional prefix prepended to captured HTTP response fields. A null value (default) won't add any prefix.
      Returns:
      this.

    • setForceContentTypeDetection

      public HttpClientFetcherConfig setForceContentTypeDetection(boolean forceContentTypeDetection)
      Whether content type is detected instead of relying on returned Content-Type HTTP response header.
      Returns:
      this.

    • setForceCharsetDetection

      public HttpClientFetcherConfig setForceCharsetDetection(boolean forceCharsetDetection)
      Whether character encoding is detected instead of relying on the charset sometimes found in the Content-Type HTTP response header.
      Returns:
      this.

    • setAuthentication

      public HttpClientFetcherConfig setAuthentication(HttpAuthConfig authentication)
      Authentication configuration for sites requiring it. Default is null.
      Returns:
      this.

    • setCookieSpec

      public HttpClientFetcherConfig setCookieSpec(HttpClientFetcherConfig.CookieSpec cookieSpec)
      Cookie specification to use when fetching documents. Default is relaxed.
      Returns:
      this.

    • setConnectionTimeout

      public HttpClientFetcherConfig setConnectionTimeout(Duration connectionTimeout)
      The connection timeout for a connection to be established. Default is DEFAULT_TIMEOUT.
      Returns:
      this.

    • setSocketTimeout

      public HttpClientFetcherConfig setSocketTimeout(Duration socketTimeout)
      Gets the maximum period of inactivity between two consecutive data packets. Default is DEFAULT_TIMEOUT.
      Returns:
      this.

    • setConnectionRequestTimeout

      public HttpClientFetcherConfig setConnectionRequestTimeout(Duration connectionRequestTimeout)
      Gets the timeout when requesting a connection. Default is DEFAULT_TIMEOUT.
      Returns:
      this.

    • setLocalAddress

      public HttpClientFetcherConfig setLocalAddress(String localAddress)
      The local address, which may be useful when working with multiple network interfaces.
      Returns:
      this.

    • setExpectContinueEnabled

      public HttpClientFetcherConfig setExpectContinueEnabled(boolean expectContinueEnabled)
      Whether 'Expect: 100-continue' handshake is enabled. See RequestConfig.isExpectContinueEnabled()
      Returns:
      this.

    • setMaxRedirects

      public HttpClientFetcherConfig setMaxRedirects(int maxRedirects)
      The maximum number of redirects to be followed. This can help prevent infinite loops. A value of zero effectively disables redirects. Default is DEFAULT_MAX_REDIRECT.
      Returns:
      this.

    • setMaxConnections

      public HttpClientFetcherConfig setMaxConnections(int maxConnections)
      The maximum number of connections that can be created. Typically, you would have at least the same amount as threads. Default is DEFAULT_MAX_CONNECTIONS.
      Returns:
      this.

    • setMaxConnectionsPerRoute

      public HttpClientFetcherConfig setMaxConnectionsPerRoute(int maxConnectionsPerRoute)
      The maximum number of connections that can be used per route. Default is DEFAULT_MAX_CONNECTIONS_PER_ROUTE.
      Returns:
      this.

    • setMaxConnectionIdleTime

      public HttpClientFetcherConfig setMaxConnectionIdleTime(Duration maxConnectionIdleTime)
      Sets the period of time after which to evict idle connections from the connection pool. Default is DEFAULT_MAX_IDLE_TIME.
      Returns:
      this.

    • setMaxConnectionInactiveTime

      public HttpClientFetcherConfig setMaxConnectionInactiveTime(Duration maxConnectionInactiveTime)
      Sets the period of time a connection must be inactive to be checked in case it became stalled. Default is 0 (not pro-actively checked).
      Returns:
      this.

    • setIfModifiedSinceDisabled

      public HttpClientFetcherConfig setIfModifiedSinceDisabled(boolean ifModifiedSinceDisabled)
      Whether adding the If-Modified-Since HTTP request header is disabled. Servers supporting this header will only return the requested document if it was last modified since the supplied date.
      Returns:
      this.

    • setETagDisabled

      public HttpClientFetcherConfig setETagDisabled(boolean eTagDisabled)
      Whether adding "ETag" If-None-Match HTTP request header is disabled. Servers supporting this header will only return the requested document if the ETag value has changed, indicating a more recent version is available.
      Returns:
      this.

    • setUserAgent

      public HttpClientFetcherConfig setUserAgent(String userAgent)
      The user-agent used when identifying the crawler to targeted web sites. It is highly recommended to always identify yourself.
      Returns:
      this.

    • setRedirectUrlProvider

      public HttpClientFetcherConfig setRedirectUrlProvider(RedirectUrlProvider redirectUrlProvider)
      The redirect URL provider. Defaults to GenericRedirectUrlProvider.
      Returns:
      this.

    • setTrustAllSSLCertificates

      public HttpClientFetcherConfig setTrustAllSSLCertificates(boolean trustAllSSLCertificates)
      Sets whether to trust all SSL certificate (affects only "https" connections). This is typically a bad idea if you care to avoid "man-in-the-middle" attacks. Try to install a SSL certificate locally to ensure a proper certificate exchange instead.
      Returns:
      this.
      Since:
      1.3.0

    • setSniDisabled

      public HttpClientFetcherConfig setSniDisabled(boolean sniDisabled)
      Sets whether Server Name Indication (SNI) is disabled.
      Returns:
      this.

    • setHstsDisabled

      public HttpClientFetcherConfig setHstsDisabled(boolean hstsDisabled)
      Gets whether the forcing of non secure URLs to secure ones is disabled, according to the URL domain Strict-Transport-Security policy (obtained from HTTP response header).
      Returns:
      this.

    • equals

      public boolean equals(Object o)
      Overrides:
      equals in class BaseFetcherConfig

    • canEqual

      protected boolean canEqual(Object other)
      Overrides:
      canEqual in class BaseFetcherConfig

    • hashCode

      public int hashCode()
      Overrides:
      hashCode in class BaseFetcherConfig

    • toString

      public String toString()
      Overrides:
      toString in class BaseFetcherConfig