Package com.norconex.crawler.web.doc.operations.delay.impl

Class AbstractDelayResolver<T extends BaseDelayResolverConfig>

java.lang.Object
com.norconex.crawler.web.doc.operations.delay.impl.AbstractDelayResolver<T>
Type Parameters:
T - type of configuration class
All Implemented Interfaces:
Configurable<T>, DelayResolver
Direct Known Subclasses:
GenericDelayResolver, ReferenceDelayResolver
public abstract class AbstractDelayResolver<T extends BaseDelayResolverConfig> extends Object implements DelayResolver, Configurable<T>

Base implementation for creating voluntary delays between URL downloads. This base class offers a few ways the actual delay value can be defined (in order):

  1. Takes the delay specify by a robots.txt file. Only applicable if robots.txt files and its robots crawl delays are not ignored.
  2. Takes an explicitly specified delay, as per implementing class.
  3. Use the specified default delay or 3 seconds, if none is specified.

One of these following scope dictates how the delay is applied, listed in order from the best behaved to the least.

  • crawler: the delay is applied between each URL download within a crawler instance, regardless how many threads are defined within that crawler, or whether URLs are from the same site or not. This is the default scope.
  • site: the delay is applied between each URL download from the same site within a crawler instance, regardless how many threads are defined. A site is defined by a URL protocol and its domain (e.g. http://example.com).
  • thread: the delay is applied between each URL download from any given thread. The more threads you have the less of an impact the delay will have.
Since:
2.5.0

  • Constructor Details

    • AbstractDelayResolver

      protected AbstractDelayResolver()

  • Method Details

    • delay

      public void delay(RobotsTxt robotsTxt, String url)
      Description copied from interface: DelayResolver
      Delay crawling activities (if applicable).
      Specified by:
      delay in interface DelayResolver
      Parameters:
      robotsTxt - robots.txt instance (if applicable)
      url - the URL being crawled

    • resolveExplicitDelay

      protected abstract Duration resolveExplicitDelay(String url)
      Resolves explicitly specified delay, in milliseconds. This method is only invoked when there are no delays from robots.txt. If the implementing class does not have a delay resolution, -1 or null is returned (the default delay will be used).
      Parameters:
      url - URL for which to resolve delay
      Returns:
      delay in millisecond, or -1 or null

    • equals

      public boolean equals(Object o)
      Overrides:
      equals in class Object

    • canEqual

      protected boolean canEqual(Object other)

    • hashCode

      public int hashCode()
      Overrides:
      hashCode in class Object

    • toString

      public String toString()
      Overrides:
      toString in class Object