Package com.norconex.crawler.web.doc.operations.delay.impl

Class BaseDelayResolverConfig

java.lang.Object
com.norconex.crawler.web.doc.operations.delay.impl.BaseDelayResolverConfig
Direct Known Subclasses:
GenericDelayResolverConfig, ReferenceDelayResolverConfig
public class BaseDelayResolverConfig extends Object

Base implementation for creating voluntary delays between URL downloads. This base class offers a few ways the actual delay value can be defined (in order):

  1. Takes the delay specify by a robots.txt file. Only applicable if robots.txt files and its robots crawl delays are not ignored.
  2. Takes an explicitly specified delay, as per implementing class.
  3. Use the specified default delay or 3 seconds, if none is specified.

One of these following scope dictates how the delay is applied, listed in order from the best behaved to the least.

  • CRAWLER: the delay is applied between each URL download within a crawler instance, regardless how many threads are defined within that crawler, or whether URLs are from the same site or not. This is the default scope.
  • SITE: the delay is applied between each URL download from the same site within a crawler instance, regardless how many threads are defined. A site is defined by a URL protocol and its domain (e.g. http://example.com).
  • THREAD: the delay is applied between each URL download from any given thread. The more threads you have the less of an impact the delay will have.
Since:
2.5.0

  • Field Details

  • Constructor Details

    • BaseDelayResolverConfig

      public BaseDelayResolverConfig()

  • Method Details

    • getDefaultDelay

      public Duration getDefaultDelay()
      The default delay.

    • isIgnoreRobotsCrawlDelay

      public boolean isIgnoreRobotsCrawlDelay()
      Whether to ignore crawl delays specified in a site robots.txt file. Not applicable when robots.txt are ignored.

    • getScope

      public BaseDelayResolverConfig.DelayResolverScope getScope()
      Gets the delay scope. See class documentation for a description of supported scopes.

    • setDefaultDelay

      public BaseDelayResolverConfig setDefaultDelay(Duration defaultDelay)
      The default delay.
      Returns:
      this.

    • setIgnoreRobotsCrawlDelay

      public BaseDelayResolverConfig setIgnoreRobotsCrawlDelay(boolean ignoreRobotsCrawlDelay)
      Whether to ignore crawl delays specified in a site robots.txt file. Not applicable when robots.txt are ignored.
      Returns:
      this.

    • setScope

      public BaseDelayResolverConfig setScope(BaseDelayResolverConfig.DelayResolverScope scope)
      Gets the delay scope. See class documentation for a description of supported scopes.
      Returns:
      this.

    • equals

      public boolean equals(Object o)
      Overrides:
      equals in class Object

    • canEqual

      protected boolean canEqual(Object other)

    • hashCode

      public int hashCode()
      Overrides:
      hashCode in class Object

    • toString

      public String toString()
      Overrides:
      toString in class Object