Package com.norconex.crawler.web.fetch.util

Class ApacheHttpUtil

java.lang.Object
com.norconex.crawler.web.fetch.util.ApacheHttpUtil
public final class ApacheHttpUtil extends Object
Utility methods for fetcher implementations using Apache HttpClient.
Since:
3.0.0

  • Method Details

    • applyResponseContent

      public static boolean applyResponseContent(org.apache.hc.core5.http.ClassicHttpResponse response, Doc doc) throws IOException

      Applies the HTTP response content to a document if such content exists. The stream is fully downloaded and associated with a document.

      Parameters:
      response - the HTTP response
      doc - document to apply headers on
      Returns:
      true if there was content to apply
      Throws:
      IOException - could not read existing content

    • applyResponseHeaders

      public static void applyResponseHeaders(org.apache.hc.core5.http.HttpResponse response, String prefix, Doc doc, WebCrawlerEntry crawlEntry)

      Applies the HTTP response headers to a document. This method will do its best to derive relevant information from the HTTP headers that can be set on the document WebCrawlerEntry:

      • Content type
      • Content encoding
      • ETag

      In addition, all HTTP headers will be added to the document metadata, with an optional prefix.

      Parameters:
      response - the HTTP response
      prefix - optional metadata prefix for all HTTP response headers
      doc - document to apply headers on
      crawlEntry - crawl ledger entry

    • applyContentTypeAndCharset

      public static void applyContentTypeAndCharset(String value, Doc doc)
      Applies the Content-Type HTTP response header on the supplied document info. It does so by extracting both the content type and charset from the value, and sets them by invoking Doc.setContentType(ContentType) and Doc.setCharset(Charset). This method is automatically invoked by applyResponseHeaders(HttpResponse, String, Doc, WebCrawlerEntry) when encountering a content type header.
      Parameters:
      value - value to parse and set.
      doc - document info

    • setRequestIfModifiedSince

      public static void setRequestIfModifiedSince(org.apache.hc.core5.http.HttpRequest request, CrawlerDocContext docCtx)
      Sets the If-Modified-Since HTTP request header based on document cached last crawled date (if any).
      Parameters:
      request - HTTP request
      docCtx - document context

    • setRequestIfNoneMatch

      public static void setRequestIfNoneMatch(org.apache.hc.core5.http.HttpRequest request, CrawlerDocContext docCtx)
      Sets the ETag If-None-Match HTTP request header based on document cached ETag value (if any).
      Parameters:
      request - HTTP request
      docCtx - document

    • createUriRequest

      public static org.apache.hc.client5.http.classic.methods.HttpUriRequestBase createUriRequest(String url, String method)
      Creates an HTTP request.
      Parameters:
      url - the request target URL
      method - HTTP method (defaults to GET if null)
      Returns:
      Apache HTTP request

    • createUriRequest

      public static org.apache.hc.client5.http.classic.methods.HttpUriRequestBase createUriRequest(String url, HttpMethod method)
      Creates an HTTP request.
      Parameters:
      url - the request target URL
      method - HTTP method (defaults to GET if null)
      Returns:
      Apache HTTP request

    • authenticateUsingForm

      public static void authenticateUsingForm(org.apache.hc.client5.http.classic.HttpClient httpClient, HttpAuthConfig authConfig) throws IOException, URISyntaxException
      Throws:
      IOException
      URISyntaxException