Package com.norconex.crawler.web.doc.operations.delay.impl

Class GenericDelayResolver

java.lang.Object
com.norconex.crawler.web.doc.operations.delay.impl.AbstractDelayResolver<GenericDelayResolverConfig>
com.norconex.crawler.web.doc.operations.delay.impl.GenericDelayResolver
All Implemented Interfaces:
Configurable<GenericDelayResolverConfig>, EventListener<Event>, DelayResolver, EventListener, Consumer<Event>
public class GenericDelayResolver extends AbstractDelayResolver<GenericDelayResolverConfig> implements EventListener<Event>

Default implementation for creating voluntary delays between URL downloads. There are a few ways the actual delay value can be defined (in order):

  1. Takes the delay specify by a robots.txt file. Only applicable if robots.txt files and its robots crawl delays are not ignored.
  2. Takes an explicitly scheduled delay, if any (picks the first one matching).
  3. Use the specified default delay or 3 seconds, if none is specified.

In a delay schedule, the days of weeks are spelled out (in English): Monday, Tuesday, etc. Time ranges are using the 24h format.

One of these following scope dictates how the delay is applied, listed in order from the best behaved to the least.

  • crawler: the delay is applied between each URL download within a crawler instance, regardless how many threads are defined within that crawler, or whether URLs are from the same site or not. This is the default scope.
  • site: the delay is applied between each URL download from the same site within a crawler instance, regardless how many threads are defined. A site is defined by a URL protocol and its domain (e.g. http://example.com).
  • thread: the delay is applied between each URL download from any given thread. The more threads you have the less of an impact the delay will have.