Package com.norconex.crawler.web.doc.operations.delay.impl

Class ReferenceDelayResolver

java.lang.Object

com.norconex.crawler.web.doc.operations.delay.impl.AbstractDelayResolver<ReferenceDelayResolverConfig>

com.norconex.crawler.web.doc.operations.delay.impl.ReferenceDelayResolver

All Implemented Interfaces:

Configurable<ReferenceDelayResolverConfig>, DelayResolver

public class ReferenceDelayResolver
extends AbstractDelayResolver<ReferenceDelayResolverConfig>

Introduces different delays between document downloads based on matching document reference (URL) patterns. There are a few ways the actual delay value can be defined (in order):

1. Takes the delay specify by a robots.txt file. Only applicable if robots.txt files and its robots crawl delays are not ignored.
2. Takes the delay matching a reference pattern, if any (picks the first one matching).
3. Used the specified default delay or 3 seconds, if none is specified.

One of these following scope dictates how the delay is applied, listed in order from the best behaved to the least.

• crawler: the delay is applied between each URL download within a crawler instance, regardless how many threads are defined within that crawler, or whether URLs are from the same site or not. This is the default scope.
• site: the delay is applied between each URL download from the same site within a crawler instance, regardless how many threads are defined. A site is defined by a URL protocol and its domain (e.g. http://example.com).
• thread: the delay is applied between each URL download from any given thread. The more threads you have the less of an impact the delay will have.

Since:

2.5.0

Field Summary

Fields inherited from interface com.norconex.commons.lang.config.Configurable

PROPERTY

Constructor Summary

Constructors

Constructor	Description
ReferenceDelayResolver()

Method Summary

Modifier and Type	Method	Description
protected boolean	canEqual(Object other)
boolean	equals(Object o)
ReferenceDelayResolverConfig	getConfiguration()
int	hashCode()
protected Duration	resolveExplicitDelay(String url)	Resolves explicitly specified delay, in milliseconds.
String	toString()

Methods inherited from class com.norconex.crawler.web.doc.operations.delay.impl.AbstractDelayResolver

delay

Methods inherited from class java.lang.Object

clone, finalize, getClass, notify, notifyAll, wait, wait, wait

Constructor Details

ReferenceDelayResolver

public ReferenceDelayResolver()

Method Details

resolveExplicitDelay

protected Duration resolveExplicitDelay(String url)

Description copied from class: AbstractDelayResolver

Resolves explicitly specified delay, in milliseconds. This method is only invoked when there are no delays from robots.txt. If the implementing class does not have a delay resolution, -1 or null is returned (the default delay will be used).

Specified by:

resolveExplicitDelay in class AbstractDelayResolver<ReferenceDelayResolverConfig>

Parameters:

url - URL for which to resolve delay

Returns:

delay in millisecond, or -1 or null

equals

public boolean equals(Object o)

Overrides:

equals in class AbstractDelayResolver<ReferenceDelayResolverConfig>

canEqual

protected boolean canEqual(Object other)

Overrides:

canEqual in class AbstractDelayResolver<ReferenceDelayResolverConfig>

hashCode

public int hashCode()

Overrides:

hashCode in class AbstractDelayResolver<ReferenceDelayResolverConfig>

toString

public String toString()

Overrides:

toString in class AbstractDelayResolver<ReferenceDelayResolverConfig>

getConfiguration

public ReferenceDelayResolverConfig getConfiguration()