Package com.norconex.crawler.web.fetch.impl.httpclient

Class HttpClientFetcher

java.lang.Object
com.norconex.crawler.core.fetch.AbstractFetcher<HttpClientFetcherConfig>
com.norconex.crawler.web.fetch.impl.httpclient.HttpClientFetcher
All Implemented Interfaces:
Configurable<HttpClientFetcherConfig>, EventListener<Event>, Fetcher, EventListener, Consumer<Event>
public class HttpClientFetcher extends AbstractFetcher<HttpClientFetcherConfig>

Default web crawler implementation of Fetcher, based on Apache HttpClient.

The "validStatusCodes" and "notFoundStatusCodes" configuration options expect a comma-separated list of HTTP response codes. If a code is added to both, the valid list takes precedence.

Accepted HTTP methods

By default this fetcher accepts HTTP GET and HEAD requests. You can limit it to only process one method with HttpClientFetcherConfig.setHttpMethods(List).

Content type and character encoding

The default way for the Web Crawler to identify the content type and character encoding of a document is to rely on the "Content-Type" HTTP response header. Web servers can sometimes return invalid or missing content type and character encoding information. You can optionally decide not to trust web servers HTTP responses and have the collector perform its own content type and encoding detection. Such detection can be enabled with HttpClientFetcherConfig.setForceContentTypeDetection(boolean) and HttpClientFetcherConfig.setForceCharsetDetection(boolean).

HSTS Support

Upon first encountering a secure site, this fetcher will check whether the site root domain has the "Strict-Transport-Security" (HSTS) policy support part of its HTTP response headers. That information gets cached for future requests. If the site supports HSTS, any non-secure URLs encountered on the same domain will be automatically converted to "https" (including sub-domains if HSTS indicates as such).

If you want to convert non-secure URLs secure ones regardless of website HSTS support, use GenericUrlNormalizerConfig.Normalization.SECURE_SCHEME instead. To disable HSTS support, use HttpClientFetcherConfig.setHstsDisabled(boolean).

Pro-active change detection

This fetcher takes advantage of the ETag and If-Modified-Since HTTP specifications.

On subsequent crawls, HTTP requests will include previously cached ETag and If-Modified-Since values to tell supporting servers we only want to download a document if it was modified since our last request. To disable support for pro-active change detection, you can use HttpClientFetcherConfig.setIfModifiedSinceDisabled(boolean) and HttpClientFetcherConfig.setETagDisabled(boolean).

These settings have no effect for web servers not supporting them.

Since:
3.0.0 (Merged from GenericDocumentFetcher and GenericHttpClientFactory)

  • Constructor Details

    • HttpClientFetcher

      public HttpClientFetcher()

  • Method Details

    • fetch

      public WebFetchResponse fetch(FetchRequest fetchRequest) throws FetchException
      Throws:
      FetchException

    • acceptRequest

      protected boolean acceptRequest(@NonNull @NonNull FetchRequest fetchRequest)
      Description copied from class: AbstractFetcher
      Optional method for subclasses to override when in need for additional acceptance logic beyond reference filters (which this abstract class provides). This method is only invoked if there are no reference filters configured on this class, or all of them accepts the request document. Default implementation does nothing (always return true).
      Overrides:
      acceptRequest in class AbstractFetcher<HttpClientFetcherConfig>
      Parameters:
      fetchRequest - the fetch request to evaluate (never null)
      Returns:
      true if accepted

    • getHttpClient

      public org.apache.hc.client5.http.classic.HttpClient getHttpClient()

    • fetcherStartup

      protected void fetcherStartup(CrawlerSession crawler)
      Description copied from class: AbstractFetcher
      Invoked once per fetcher instance, when the crawler starts. Default implementation does nothing.
      Overrides:
      fetcherStartup in class AbstractFetcher<HttpClientFetcherConfig>
      Parameters:
      crawler - crawler

    • fetcherShutdown

      protected void fetcherShutdown(CrawlerSession c)
      Description copied from class: AbstractFetcher
      Invoked once per fetcher when the crawler ends. Default implementation does nothing.
      Overrides:
      fetcherShutdown in class AbstractFetcher<HttpClientFetcherConfig>
      Parameters:
      c - crawler

    • getUserAgent

      public String getUserAgent()

    • createHttpClient

      protected org.apache.hc.client5.http.classic.HttpClient createHttpClient()

    • createConnectionManager

      protected org.apache.hc.client5.http.io.HttpClientConnectionManager createConnectionManager()

    • createRoutePlanner

      protected org.apache.hc.client5.http.routing.HttpRoutePlanner createRoutePlanner(org.apache.hc.client5.http.SchemePortResolver schemePortResolver)

    • buildCustomHttpClient

      protected void buildCustomHttpClient(org.apache.hc.client5.http.impl.classic.HttpClientBuilder builder)
      For implementors to subclass. Does nothing by default.
      Parameters:
      builder - http client builder

    • authenticateUsingForm

      protected void authenticateUsingForm(org.apache.hc.client5.http.classic.HttpClient httpClient)

    • createDefaultCookieStore

      protected org.apache.hc.client5.http.cookie.CookieStore createDefaultCookieStore()
      Creates the default cookie store to be added to each request context.
      Returns:
      a cookie store

    • createDefaultRequestHeaders

      protected List<org.apache.hc.core5.http.Header> createDefaultRequestHeaders()

      Creates a list of HTTP headers based on configuration.

      This method will also add a "Basic" authentication header if "preemptive" is true on the authentication configuration and credentials were supplied.

      Returns:
      a list of HTTP request headers

    • createSchemePortResolver

      protected org.apache.hc.client5.http.SchemePortResolver createSchemePortResolver()

    • createRequestConfig

      protected org.apache.hc.client5.http.config.RequestConfig createRequestConfig()

    • createProxy

      protected org.apache.hc.core5.http.HttpHost createProxy()

    • createCredentialsProvider

      protected org.apache.hc.client5.http.auth.CredentialsProvider createCredentialsProvider()

    • createConnectionConfig

      protected org.apache.hc.client5.http.config.ConnectionConfig createConnectionConfig()

    • createTlsSocketStrategy

      protected org.apache.hc.client5.http.ssl.TlsSocketStrategy createTlsSocketStrategy(SSLContext sslContext)

    • createSSLContext

      protected SSLContext createSSLContext()

    • equals

      public boolean equals(Object o)
      Overrides:
      equals in class AbstractFetcher<HttpClientFetcherConfig>

    • canEqual

      protected boolean canEqual(Object other)
      Overrides:
      canEqual in class AbstractFetcher<HttpClientFetcherConfig>

    • hashCode

      public int hashCode()
      Overrides:
      hashCode in class AbstractFetcher<HttpClientFetcherConfig>

    • toString

      public String toString()
      Overrides:
      toString in class AbstractFetcher<HttpClientFetcherConfig>

    • getConfiguration

      public HttpClientFetcherConfig getConfiguration()