Package com.norconex.crawler.core

Class CrawlerCallbacks

java.lang.Object

com.norconex.crawler.core.CrawlerCallbacks

All Implemented Interfaces:

EventListener<Event>, EventListener, Consumer<Event>

public class CrawlerCallbacks
extends Object
implements EventListener<Event>

Optional callbacks invoked at specific times in the crawler life-cycle. The flow of execution for a crawl is as follow (callbacks are bold):

• Launch crawler.
• Invoke beforeSession.
• Initialize crawl session.
• For each command:
  • Invoke beforeCommand.
  • Execute command.
  • For each document:
    • Invoke beforeDocumentProcessing.
    • Process document (fetch, parse, transform, etc.)
    • Invoke afterDocumentProcessing.
    • Invoke beforeDocumentFinalizing.
    • Finalize document (upsert/delete, close resources, etc.)
    • Invoke afterDocumentFinalizing.
  • Invoke afterCommand
• Invoke afterSession

Nested Class Summary

Nested Classes

Modifier and Type	Class	Description
static class	CrawlerCallbacks.CrawlerCallbacksBuilder
static interface	CrawlerCallbacks.CrawlerCommandCallback	Convenience interface providing a default implementation of execute that ignores all commands but CrawlerCommand and offers an overloaded accept method with a single argument.

Method Summary

Modifier and Type	Method	Description
void	accept(Event event)	Dispatches crawler lifecycle events to the appropriate hook.
static CrawlerCallbacks.CrawlerCallbacksBuilder	builder()
BiConsumer<CrawlerSession,Class<? extends Command>>	getAfterCommand()	Invoked after a command has been executed, but before resources are closed.
BiConsumer<CrawlerSession,Doc>	getAfterDocumentFinalizing()
BiConsumer<CrawlerSession,Doc>	getAfterDocumentProcessing()
Consumer<CrawlerConfig>	getAfterSession()	Invoked after a crawl session has been destroyed and resources closed.
BiConsumer<CrawlerSession,Class<? extends Command>>	getBeforeCommand()	Invoked before a command is executed, after the session and crawl context have been initialized.
BiConsumer<CrawlerSession,Doc>	getBeforeDocumentFinalizing()
BiConsumer<CrawlerSession,Doc>	getBeforeDocumentProcessing()
Consumer<CrawlerConfig>	getBeforeSession()	Invoked before the crawl session gets initialized (which is also before any command gets executed).

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Methods inherited from interface java.util.function.Consumer

andThen

Method Details

accept

public void accept(Event event)

Dispatches crawler lifecycle events to the appropriate hook. This method is called automatically when this instance is registered as an EventListener on the crawler's EventManager.

Specified by:

accept in interface Consumer<Event>

builder

public static CrawlerCallbacks.CrawlerCallbacksBuilder builder()

getBeforeSession

public Consumer<CrawlerConfig> getBeforeSession()

Invoked before the crawl session gets initialized (which is also before any command gets executed). Modifying the CrawlerConfig is generally safe here.

getAfterSession

public Consumer<CrawlerConfig> getAfterSession()

Invoked after a crawl session has been destroyed and resources closed.

getBeforeCommand

public BiConsumer<CrawlerSession,Class<? extends Command>> getBeforeCommand()

Invoked before a command is executed, after the session and crawl context have been initialized. Rely on the supplied command class to know which command.

See Also:

• CrawlerCallbacks.CrawlerCommandCallback

getAfterCommand

public BiConsumer<CrawlerSession,Class<? extends Command>> getAfterCommand()

Invoked after a command has been executed, but before resources are closed. Rely on the supplied command class to know which command.

See Also:

• CrawlerCallbacks.CrawlerCommandCallback

getBeforeDocumentProcessing

public BiConsumer<CrawlerSession,Doc> getBeforeDocumentProcessing()

getAfterDocumentProcessing

public BiConsumer<CrawlerSession,Doc> getAfterDocumentProcessing()

getBeforeDocumentFinalizing

public BiConsumer<CrawlerSession,Doc> getBeforeDocumentFinalizing()

getAfterDocumentFinalizing

public BiConsumer<CrawlerSession,Doc> getAfterDocumentFinalizing()