Package com.norconex.crawler.web

Class WebCrawlerConfig

java.lang.Object
com.norconex.crawler.core.CrawlerConfig
com.norconex.crawler.web.WebCrawlerConfig
public class WebCrawlerConfig extends CrawlerConfig

Web Crawler configuration, adding more options to the base CrawlerConfig.

Start URLs

Crawling begins with specifying one or more references to either documents or starting points to documents you want to crawl. For a web crawl, those references are URLs (e.g., web site home page, sitemap, etc.). There are different ways to specify one or more "start" references (repeatable, shortened to "ref"):

Scope: To limit crawling to specific web domains, and avoid creating many filters to that effect, you can tell the crawler to "stay" within the web site "scope" with setUrlScopeResolver(UrlScopeResolver).

URL Normalization

Pages on web sites are often referenced using different URL patterns. Such URL variations can fool the crawler into downloading the same document multiple times. To avoid this, URLs are "normalized". That is, they are converted so they are always formulated the same way. By default, the crawler only applies normalization in ways that are semantically equivalent (see GenericUrlNormalizer).

Crawl Speed

Be kind to web sites you crawl. Being too aggressive can be perceived as a cyber-attack by the targeted web site (e.g., DoS attack). This can lead to your crawler being blocked.

For this reason, the crawler plays nice by default. It will wait a few seconds between each page download, regardless of the maximum number of threads specified or whether pages crawled are on different web sites. This can of course be changed to be as fast as you want. See GenericDelayResolver) for changing default options. You can also provide your own "delay resolver" by supplying a class implementing DelayResolver.

Crawl Depth

The crawl depth represents how many level from the start URL the crawler goes. From a browser user perspective, it can be seen as the number of link "clicks" required from a start URL in order to get to a specific page. The crawler will crawl as deep for as long as it discovers new URLs not getting rejected by your configuration. This is not always desirable. For instance, a web site could have dynamically generated URLs with infinite possibilities (e.g., dynamically generated web calendars). To avoid infinite crawls, it is recommended to limit the maximum depth to something reasonable for your site with CrawlerConfig.setMaxDepth(int).

By default the crawler stores, as metadata, URLs extracted from documents that are in scope. Exceptions are pages discovered at the configured maximum depth (CrawlerConfig.setMaxDepth(int)). This can be changed using the setKeepReferencedLinks(Set) method. Changing this setting has no incidence on what page gets crawled. Possible options are:

Orphan documents

Orphans are valid documents, which on subsequent crawls can no longer be reached (e.g. there are no longer referenced). This is regardless whether the file has been deleted or not at the source. You can tell the crawler how to handle those with CrawlerConfig.setOrphansStrategy(OrphansStrategy). Possible options are:

  • PROCESS: Default. Tries to crawl orphans normally as if they were still reachable by the crawler.
  • IGNORE: Does nothing with orphans (not deleted, not processed)..
  • DELETE: Orphans are sent to your Committer for deletion.

Error Handling

By default the crawler logs exceptions while trying to prevent them from terminating a crawling session. There might be cases where you want the crawler to halt upon encountering some types of exceptions. You can do so with CrawlerConfig.setStopOnExceptions(List).

Crawler Events

The crawler fires all kind of events to notify interested parties of such things as when a document is rejected, imported, committed, etc.). You can listen to crawler events using CrawlerConfig.setEventListeners(List).

Document Importing

The process of transforming, enhancing, parsing to extracting plain text and many other document-specific processing activities are handled by the Norconex Importer module. See ImporterConfig for many additional configuration options.

Bad Documents

On a fresh crawl, documents that are unreachable or not obtained successfully for some reason are simply logged and ignored. On the other hand, documents that were successfully crawled once and are suddenly failing on a subsequent crawl are considered "spoiled". You can decide whether to grace (retry next time), delete, or ignore those spoiled documents with CrawlerConfig.setSpoiledReferenceStrategizer(SpoiledReferenceStrategizer).

Committing Documents

The last step of a successful processing of a document is to store it in your preferred target repository (or repositories). For this to happen, you have to configure one or more Committers corresponding to your needs or create a custom one. You can have a look at available Committers here: https://opensource.norconex.com/committers/ See CrawlerConfig.setCommitters(List).

HTTP Fetcher

To crawl and parse a document, it first needs to be downloaded. This is the role of one or more HTTP Fetchers. HttpClientFetcher is the default implementation and can handle most web sites. There might be cases where a more specialized way of obtaining web resources is needed. For instance, JavaScript-generated web pages are often best handled by web browsers. In such case you can use the WebDriverFetcher. You can also use CrawlerConfig.setFetchers(List) to supply your own fetcher implementation.

HTTP Methods

A fetcher typically issues an HTTP GET request to obtain a document. There might be cases where you first want to issue a separate HEAD request. One example is to filter documents based on the HTTP HEAD response information, thus possibly saving downloading large files you don't want.

You can tell the crawler how it should handle HTTP GET and HEAD requests using using CrawlerConfig.setDocumentFetchSupport(FetchDirectiveSupport) and CrawlerConfig.setMetadataFetchSupport(FetchDirectiveSupport) respectively. For each, the options are:

  • DISABLED: No HTTP call will be made using that method.
  • OPTIONAL: If the HTTP method is not supported by any fetcher or the HTTP request for it was not successful, the document can still be processed successfully by the other HTTP method. Only relevant when both HEAD and GET are enabled.
  • REQUIRED: If the HTTP method is not supported by any fetcher or the HTTP request for it was not successful, the document will be rejected and won't go any further, even if the other HTTP method was or could have been successful. Only relevant when both HEAD and GET are enabled.

If you enable only one HTTP method (default), then specifying OPTIONAL or REQUIRED for it have the same effect. At least one method needs to be enabled for an HTTP request to be attempted. By default HEAD requests are DISABLED and GET are REQUIRED. If you are unsure what settings to use, keep the defaults.

Filtering Unwanted Documents

Without filtering, you would typically crawl many documents you are not interested in. There are different types filtering offered to you, occurring at different time during a URL crawling process. The sooner in a URL processing life-cycle you filter out a document the more you can improve the crawler performance. It may be important for you to understand the differences:

  • Reference filters: The fastest way to exclude a document. The filtering rule applies on the URL, before any HTTP request is made for that URL. Rejected documents are not queued for processing. They are not downloaded (thus no URLs are extracted). The specified "delay" between downloads is not applied (i.e. no delay for rejected documents).

  • Metadata filters: Applies filtering on a document metadata fields.

    If CrawlerConfig.getMetadataFetchSupport() value forces a distinct call for fetching metadata, these filters will be invoked after the crawler performs an HTTP HEAD request. It gives you the opportunity to filter documents based on the HTTP HEAD response to potentially save a more expensive HTTP GET request for download (but results in two HTTP requests for valid documents -- HEAD and GET). Filtering occurs before URLs are extracted (since no content is downloaded.

    When CrawlerConfig.getMetadataFetchSupport() does not invoke making a distinct call for metadata, these filters will be invoked on the metadata of the HTTP response obtained from an HTTP GET request (as the document is downloaded). Filtering occurs after URLs are extracted.

  • Document filters: Use when having access to the document itself (and its content) is required to apply filtering. Always triggered after a document is downloaded and after URLs are extracted, but before it is imported (Importer module).
  • Importer filters: The Importer module also offers document filtering options. At that point a document is already downloaded and its links extracted. There are two types of filtering offered by the Importer: before and after document parsing (assuming you configured at least one parser). Use filters before parsing if you need to filter on raw content or want to avoid parsing some documents. Use filters after parsing when you need to read the content as plain text.

Robot Directives

By default, the crawler tries to respect instructions a web site has put in place for the benefit of crawlers. The following is a list of some of the popular ones. Where null can be set to disable support for specific instructions, you can achieve the equivalent in XML configuration by declaring the corresponding option as a self-closed tag.

Re-crawl Frequency

The crawler will crawl any given URL at most one time per crawling session. It is possible to skip documents that are not yet "ready" to be re-crawled to speed up each crawling sessions. Sitemap.xml directives to that effect are respected by default ("frequency" and "lastmod"). You can have your own conditions for re-crawl with setRecrawlableResolver(RecrawlableResolver). This feature can be used for instance, to crawl a "news" section of your site more frequently than let's say, an "archive" section of your site.

Change Detection (Checksums)

To find out if a document has changed from one crawling session to another, the crawler creates and keeps a digital signature, or checksum of each crawled documents. Upon crawling the same URL again, a new checksum is created and compared against the previous one. Any difference indicates a modified document. There are two checksums at play, tested at different times. One obtained from a document metadata (default is LastModifiedMetadataChecksummer, and one from the document itself Md5DocumentChecksummer. You can provide your own implementation. See: CrawlerConfig.setMetadataChecksummer(MetadataChecksummer) and CrawlerConfig.setDocumentChecksummer(DocumentChecksummer).

Deduplication

The crawler can attempt to detect and reject documents considered as duplicates within a crawler session. A document will be considered duplicate if there was already a document processed with the same metadata or document checksum. To enable this feature, set CrawlerConfig.setMetadataDeduplicate(boolean) and/or CrawlerConfig.setDocumentDeduplicate(boolean) to true. Setting those will have no effect if the corresponding checksummers are null or checksums are otherwise not are being generated.

Deduplication can impact crawl performance. It is recommended you use it only if you can't distinguish duplicates via other means (URL normalizer, canonical URL support, etc.). Also, you should only enable this feature if you know your checksummer(s) will generate a checksum that is acceptably unique to you.

URL Extraction

To be able to crawl a web site, links need to be extracted from web pages. It is the job of a link extractor. It is possible to use multiple link extractor for different type of content. By default, the HtmlLinkExtractor is used, but you can add others or provide your own with setLinkExtractors(List).

There might be cases where you want a document to be parsed by the Importer and establish which links to process yourself during the importing phase (for more advanced use cases). In such cases, you can identify a document metadata field to use as a URL holding tanks after importing has occurred. URLs in that field will become eligible for crawling. See setPostImportLinks(TextMatcher).

  • Constructor Details

    • WebCrawlerConfig

      public WebCrawlerConfig()

  • Method Details

    • getStartReferencesSitemaps

      public List<String> getStartReferencesSitemaps()
      Gets sitemap URLs to be used as starting points for crawling.
      Returns:
      sitemap URLs (never null)
      Since:
      2.3.0

    • setStartReferencesSitemaps

      public WebCrawlerConfig setStartReferencesSitemaps(List<String> startReferencesSitemaps)
      Sets the sitemap URLs used as starting points for crawling.
      Parameters:
      startReferencesSitemaps - sitemap URLs
      Returns:
      this
      Since:
      3.0.0

    • getKeepReferencedLinks

      public Set<WebCrawlerConfig.ReferencedLinkType> getKeepReferencedLinks()
      Gets what type of referenced links to keep, if any. Those links are URLs extracted by link extractors. See class documentation for more details.
      Returns:
      preferences for keeping links
      Since:
      3.0.0

    • setKeepReferencedLinks

      public WebCrawlerConfig setKeepReferencedLinks(Set<WebCrawlerConfig.ReferencedLinkType> keepReferencedLinks)
      Sets whether to keep referenced links and what to keep. Those links are URLs extracted by link extractors. See class documentation for more details.
      Parameters:
      keepReferencedLinks - option for keeping links
      Returns:
      this
      Since:
      3.0.0

    • getLinkExtractors

      public List<LinkExtractor> getLinkExtractors()
      Gets link extractors.
      Returns:
      link extractors

    • setLinkExtractors

      public WebCrawlerConfig setLinkExtractors(List<LinkExtractor> linkExtractors)
      Sets link extractors.
      Parameters:
      linkExtractors - link extractors
      Returns:
      this
      Since:
      3.0.0

    • getPostImportLinks

      public TextMatcher getPostImportLinks()
      Gets a field matcher used to identify post-import metadata fields holding URLs to consider for crawling.
      Returns:
      field matcher
      Since:
      3.0.0

    • setPostImportLinks

      public WebCrawlerConfig setPostImportLinks(TextMatcher fieldMatcher)
      Set a field matcher used to identify post-import metadata fields holding URLs to consider for crawling.
      Parameters:
      fieldMatcher - field matcher
      Returns:
      this
      Since:
      3.0.0

    • getUrlNormalizers

      public List<WebUrlNormalizer> getUrlNormalizers()
      Gets URL normalizers. Executed in the order provided. Defaults to a single GenericUrlNormalizer with its default settings.
      Returns:
      URL normalizers

    • setUrlNormalizers

      public WebCrawlerConfig setUrlNormalizers(List<WebUrlNormalizer> urlNormalizers)
      Sets URL normalizers. Executed in the order provided. Defaults to a single GenericUrlNormalizer with its default settings.
      Parameters:
      urlNormalizers - URL normalizers
      Returns:
      this

    • getUrlScopeResolver

      public UrlScopeResolver getUrlScopeResolver()
      The strategy to use to determine if a URL is in scope.

    • getDelayResolver

      public DelayResolver getDelayResolver()
      The delay resolver dictating the minimum amount of time to wait between web requests. Defaults to GenericDelayResolver.

    • getCanonicalLinkDetector

      public CanonicalLinkDetector getCanonicalLinkDetector()
      The canonical link detector. Defaults to GenericCanonicalLinkDetector. Set to null to disable canonical link detection.

    • isPostImportLinksKeep

      public boolean isPostImportLinksKeep()
      Whether to keep the Importer-populated fields from getPostImportLinks(). By default, those are deleted from a document when the URLs they contain are queued for processing or otherwise evaluated.
      See Also:

    • getRobotsTxtProvider

      public RobotsTxtProvider getRobotsTxtProvider()
      The provider of robots.txt rules for a site (if applicable). Defaults to StandardRobotsTxtProvider. Set to null to disable.

    • getRobotsMetaProvider

      public RobotsMetaProvider getRobotsMetaProvider()
      The provider of robots metadata rules for a page (if applicable). Defaults to StandardRobotsMetaProvider. Set to null to disable.

    • getSitemapResolver

      public SitemapResolver getSitemapResolver()
      The resolver of web site sitemaps (if applicable). Defaults to GenericSitemapResolver. Set to null to disable all sitemap support, or see class documentation to disable sitemap detection only.
      See Also:

    • getSitemapLocator

      public SitemapLocator getSitemapLocator()
      The locator of sitemaps (if applicable). Defaults to GenericSitemapLocator. Set to null to disable locating sitemaps (relying on sitemaps defined as start reference, if any).
      See Also:

    • getRecrawlableResolver

      public RecrawlableResolver getRecrawlableResolver()
      The resolver that indicates whether a given URL is ready to be crawled by a new crawl session. Usually amounts to checking if enough time has passed between two crawl sessions. Defaults to GenericRecrawlableResolver.

    • setUrlScopeResolver

      public WebCrawlerConfig setUrlScopeResolver(UrlScopeResolver urlScopeResolver)
      The strategy to use to determine if a URL is in scope.
      Returns:
      this.

    • setDelayResolver

      public WebCrawlerConfig setDelayResolver(DelayResolver delayResolver)
      The delay resolver dictating the minimum amount of time to wait between web requests. Defaults to GenericDelayResolver.
      Returns:
      this.

    • setCanonicalLinkDetector

      public WebCrawlerConfig setCanonicalLinkDetector(CanonicalLinkDetector canonicalLinkDetector)
      The canonical link detector. Defaults to GenericCanonicalLinkDetector. Set to null to disable canonical link detection.
      Returns:
      this.

    • setPostImportLinksKeep

      public WebCrawlerConfig setPostImportLinksKeep(boolean postImportLinksKeep)
      Whether to keep the Importer-populated fields from getPostImportLinks(). By default, those are deleted from a document when the URLs they contain are queued for processing or otherwise evaluated.
      Returns:
      this.
      See Also:

    • setRobotsTxtProvider

      public WebCrawlerConfig setRobotsTxtProvider(RobotsTxtProvider robotsTxtProvider)
      The provider of robots.txt rules for a site (if applicable). Defaults to StandardRobotsTxtProvider. Set to null to disable.
      Returns:
      this.

    • setRobotsMetaProvider

      public WebCrawlerConfig setRobotsMetaProvider(RobotsMetaProvider robotsMetaProvider)
      The provider of robots metadata rules for a page (if applicable). Defaults to StandardRobotsMetaProvider. Set to null to disable.
      Returns:
      this.

    • setSitemapResolver

      public WebCrawlerConfig setSitemapResolver(SitemapResolver sitemapResolver)
      The resolver of web site sitemaps (if applicable). Defaults to GenericSitemapResolver. Set to null to disable all sitemap support, or see class documentation to disable sitemap detection only.
      Returns:
      this.
      See Also:

    • setSitemapLocator

      public WebCrawlerConfig setSitemapLocator(SitemapLocator sitemapLocator)
      The locator of sitemaps (if applicable). Defaults to GenericSitemapLocator. Set to null to disable locating sitemaps (relying on sitemaps defined as start reference, if any).
      Returns:
      this.
      See Also:

    • setRecrawlableResolver

      public WebCrawlerConfig setRecrawlableResolver(RecrawlableResolver recrawlableResolver)
      The resolver that indicates whether a given URL is ready to be crawled by a new crawl session. Usually amounts to checking if enough time has passed between two crawl sessions. Defaults to GenericRecrawlableResolver.
      Returns:
      this.

    • equals

      public boolean equals(Object o)
      Overrides:
      equals in class CrawlerConfig

    • canEqual

      protected boolean canEqual(Object other)
      Overrides:
      canEqual in class CrawlerConfig

    • hashCode

      public int hashCode()
      Overrides:
      hashCode in class CrawlerConfig

    • toString

      public String toString()
      Overrides:
      toString in class CrawlerConfig