Package com.norconex.crawler.core.fetch

Class AbstractFetcher<C extends BaseFetcherConfig>

java.lang.Object

com.norconex.crawler.core.fetch.AbstractFetcher<C>

Type Parameters:

C - configuration type

All Implemented Interfaces:

Configurable<C>, EventListener<Event>, Fetcher, EventListener, Consumer<Event>

Direct Known Subclasses:

AbstractNioFetcher, BoxFetcher, EgnyteFetcher, GoogleDriveFetcher, HttpClientFetcher, M365GraphFetcher, PlaywrightFetcher, WebDriverFetcher

public abstract class AbstractFetcher<C extends BaseFetcherConfig>
extends Object
implements Fetcher, EventListener<Event>, Configurable<C>

Base class implementing the accept(FetchRequest) method using reference filters to determine if this fetcher will accept to fetch a document, in addition to whatever logic implementing classes may provide by optionally overriding acceptRequest(FetchRequest) (which otherwise always return true). It also offers methods to overwrite in order to react to crawler startup and shutdown events.

Field Summary

Fields inherited from interface com.norconex.commons.lang.config.Configurable

PROPERTY

Constructor Summary

Constructors

Constructor	Description
AbstractFetcher()

Method Summary

Modifier and Type	Method	Description
final void	accept(Event event)
final boolean	accept(@NonNull FetchRequest fetchRequest)
protected boolean	acceptRequest(@NonNull FetchRequest fetchRequest)	Optional method for subclasses to override when in need for additional acceptance logic beyond reference filters (which this abstract class provides).
protected boolean	canEqual(Object other)
boolean	equals(Object o)
protected void	fetcherShutdown(CrawlerSession crawler)	Invoked once per fetcher when the crawler ends.
protected void	fetcherStartup(CrawlerSession crawler)	Invoked once per fetcher instance, when the crawler starts.
int	hashCode()
String	toString()

Methods inherited from class java.lang.Object

clone, finalize, getClass, notify, notifyAll, wait, wait, wait

Methods inherited from interface com.norconex.commons.lang.config.Configurable

getConfiguration

Methods inherited from interface java.util.function.Consumer

andThen

Methods inherited from interface com.norconex.crawler.core.fetch.Fetcher

fetch

Constructor Details

AbstractFetcher

public AbstractFetcher()

Method Details

accept

public final boolean accept(@NonNull
 @NonNull FetchRequest fetchRequest)

Specified by:

accept in interface Fetcher

acceptRequest

protected boolean acceptRequest(@NonNull
 @NonNull FetchRequest fetchRequest)

Optional method for subclasses to override when in need for additional acceptance logic beyond reference filters (which this abstract class provides). This method is only invoked if there are no reference filters configured on this class, or all of them accepts the request document. Default implementation does nothing (always return true).

Parameters:

fetchRequest - the fetch request to evaluate (never null)

Returns:

true if accepted

accept

public final void accept(Event event)

Specified by:

accept in interface Consumer<C extends BaseFetcherConfig>

fetcherStartup

protected void fetcherStartup(CrawlerSession crawler)

Invoked once per fetcher instance, when the crawler starts. Default implementation does nothing.

Parameters:

crawler - crawler

fetcherShutdown

protected void fetcherShutdown(CrawlerSession crawler)

Invoked once per fetcher when the crawler ends. Default implementation does nothing.

Parameters:

crawler - crawler

equals

public boolean equals(Object o)

Overrides:

equals in class Object

canEqual

protected boolean canEqual(Object other)

hashCode

public int hashCode()

Overrides:

hashCode in class Object

toString

public String toString()

Overrides:

toString in class Object