Package com.norconex.crawler.web.doc.operations.scope.impl

Class GenericUrlScopeResolver

java.lang.Object

com.norconex.crawler.core.event.listeners.CrawlerLifeCycleListener

com.norconex.crawler.web.doc.operations.scope.impl.GenericUrlScopeResolver

All Implemented Interfaces:

Configurable<GenericUrlScopeResolverConfig>, EventListener<Event>, UrlScopeResolver, EventListener, Consumer<Event>

public class GenericUrlScopeResolver
extends CrawlerLifeCycleListener
implements UrlScopeResolver, Configurable<GenericUrlScopeResolverConfig>

By default a crawler will try to follow all links it discovers. You can define your own filters to limit the scope of the pages being crawled. When you have multiple URLs defined as start URLs, it can be tricky to perform global filtering that apply to each URLs without causing URL filtering conflicts. This class offers an easy way to address a frequent URL filtering need: to "stay on site". That is, when following a page and extracting URLs found in it, make sure to only keep URLs that are on the same site as the page URL we are on.

By default this class does not request to stay on a site.

Since:

2.3.0

Nested Class Summary

Nested classes/interfaces inherited from interface com.norconex.crawler.web.doc.operations.scope.UrlScopeResolver

UrlScopeResolver.SitemapPresence

Field Summary

Fields inherited from interface com.norconex.commons.lang.config.Configurable

PROPERTY

Fields inherited from interface com.norconex.crawler.web.doc.operations.scope.UrlScopeResolver

RESOLVED_SITES_CACHE_NAME

Constructor Summary

Constructors

Constructor	Description
GenericUrlScopeResolver()

Method Summary

Modifier and Type	Method	Description
protected boolean	canEqual(Object other)
boolean	equals(Object o)
GenericUrlScopeResolverConfig	getConfiguration()
int	hashCode()
protected void	onCrawlerCrawlBegin(CrawlerEvent event)
UrlScope	resolve(String inScopeURL, WebCrawlerEntry candidateDocContext)
String	toString()

Methods inherited from class com.norconex.crawler.core.event.listeners.CrawlerLifeCycleListener

accept, onCrawlerCleanBegin, onCrawlerCleanEnd, onCrawlerCrawlEnd, onCrawlerError, onCrawlerEvent, onCrawlerStopBegin, onCrawlerStopEnd

Methods inherited from class java.lang.Object

clone, finalize, getClass, notify, notifyAll, wait, wait, wait

Methods inherited from interface java.util.function.Consumer

andThen

Constructor Details

GenericUrlScopeResolver

public GenericUrlScopeResolver()

Method Details

onCrawlerCrawlBegin

protected void onCrawlerCrawlBegin(CrawlerEvent event)

Overrides:

onCrawlerCrawlBegin in class CrawlerLifeCycleListener

resolve

public UrlScope resolve(String inScopeURL,
 WebCrawlerEntry candidateDocContext)

Specified by:

resolve in interface UrlScopeResolver

equals

public boolean equals(Object o)

Overrides:

equals in class CrawlerLifeCycleListener

canEqual

protected boolean canEqual(Object other)

Overrides:

canEqual in class CrawlerLifeCycleListener

hashCode

public int hashCode()

Overrides:

hashCode in class CrawlerLifeCycleListener

toString

public String toString()

Overrides:

toString in class CrawlerLifeCycleListener

getConfiguration

public GenericUrlScopeResolverConfig getConfiguration()

Specified by:

getConfiguration in interface Configurable<GenericUrlScopeResolverConfig>