Package com.norconex.crawler.core.session

Class CrawlerStateStore

java.lang.Object

com.norconex.crawler.core.session.CrawlerStateStore

public class CrawlerStateStore
extends Object

Manages persistent crawl-state and session-scoped attributes stored in the distributed session cache. Can be injected directly in tests without the full CrawlerSession.

Constructor Summary

Constructors

Constructor	Description
CrawlerStateStore(CacheMap<String> sessionCache, CrawlerAttributes sessionAttributes, CrawlerAttributes runAttributes)	Creates a new CrawlerStateStore instance.

Method Summary

Modifier and Type	Method	Description
CrawlerState	getCrawlState()	Returns the current crawl state, reading fresh from the distributed cache each call.
CrawlerAttributes	getSessionAttributes()	Returns the session-scoped attribute store backed by the session cache.
boolean	isStartRefsQueueingComplete()	Returns whether start-reference queuing has completed.
CrawlerSession.State	loadState()	Loads the current crawl state directly from the distributed cache.
void	setStartRefsQueueingComplete(boolean isComplete)	Marks whether start-reference queuing has completed.
void	updateCrawlState(CrawlerState state)	Updates the crawl state in the distributed cache with a new timestamp.

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Details

CrawlerStateStore

public CrawlerStateStore(CacheMap<String> sessionCache,
 CrawlerAttributes sessionAttributes,
 CrawlerAttributes runAttributes)

Creates a new CrawlerStateStore instance.

Parameters:

sessionCache -

sessionAttributes -

runAttributes - Ephemeral (non-MapStore-backed) attributes for per-run flags that must survive partition migration after a node crash. Backed by the eph-crawlRun cache so that in-memory backup promotion is used directly without triggering a LAZY MapStore reload.

Method Details

loadState

public CrawlerSession.State loadState()

Loads the current crawl state directly from the distributed cache.

Returns:

the current state, or null if none has been saved yet

updateCrawlState

public void updateCrawlState(CrawlerState state)

Updates the crawl state in the distributed cache with a new timestamp.

Parameters:

state - the new crawl state

getCrawlState

public CrawlerState getCrawlState()

Returns the current crawl state, reading fresh from the distributed cache each call.

Returns:

the current crawl state, or null

isStartRefsQueueingComplete

public boolean isStartRefsQueueingComplete()

Returns whether start-reference queuing has completed.

Returns:

true when all start references have been queued

setStartRefsQueueingComplete

public void setStartRefsQueueingComplete(boolean isComplete)

Marks whether start-reference queuing has completed.

Parameters:

isComplete - true once all start references are queued

getSessionAttributes

public CrawlerAttributes getSessionAttributes()

Returns the session-scoped attribute store backed by the session cache.

Returns:

session attributes