Uses of Class
com.norconex.crawler.core.CrawlerConfig

Packages that use CrawlerConfig

Package	Description
com.norconex.crawler.core
com.norconex.crawler.core.cmd.stop
com.norconex.crawler.core.context
com.norconex.crawler.core.session
com.norconex.crawler.core.util
com.norconex.crawler.fs
com.norconex.crawler.web

Uses of CrawlerConfig in com.norconex.crawler.core

Methods in com.norconex.crawler.core that return CrawlerConfig

Modifier and Type	Method	Description
CrawlerConfig	CrawlerConfig.addCommitter(Committer committer)	Adds a Committer responsible for persisting information to a target location/repository.
CrawlerConfig	CrawlerConfig.addEventListener(EventListener<?> eventListener)	Adds an event listener.
CrawlerConfig	CrawlerConfig.addEventListeners(List<EventListener<?>> eventListeners)	Adds event listeners.
CrawlerConfig	CrawlerConfig.clearEventListeners()	Clears all event listeners.
CrawlerConfig	Crawler.getCrawlConfig()
CrawlerConfig	CrawlerConfig.setChangeDiscovery(CrawlerConfig.ChangeDiscovery changeDiscovery)	Change discovery strategy.
CrawlerConfig	CrawlerConfig.setClusterConfig(@NonNull ClusterConfig clusterConfig)	The Importer module configuration.
CrawlerConfig	CrawlerConfig.setCommitters(List<Committer> committers)	Sets Committers responsible for persisting information to a target location/repository.
CrawlerConfig	CrawlerConfig.setDeferredShutdownDuration(Duration deferredShutdownDuration)	The amount of time to defer the crawler shutdown when it is done executing.
CrawlerConfig	CrawlerConfig.setDocumentChecksummer(DocumentChecksummer documentChecksummer)	The document checksummer.
CrawlerConfig	CrawlerConfig.setDocumentDeduplicate(boolean documentDeduplicate)	Whether to turn ON deduplication based on document checksum.
CrawlerConfig	CrawlerConfig.setDocumentFetchSupport(FetchDirectiveSupport documentFetchSupport)
CrawlerConfig	CrawlerConfig.setDocumentFilters(List<DocumentFilter> documentFilters)	Sets document filters.
CrawlerConfig	CrawlerConfig.setEventListeners(List<EventListener<?>> eventListeners)	Sets event listeners.
CrawlerConfig	CrawlerConfig.setFetchers(List<Fetcher> fetchers)	One or more fetchers responsible for pulling documents and document metadata associated with a reference from a source.
CrawlerConfig	CrawlerConfig.setFetchersMaxRetries(int fetchersMaxRetries)	The maximum number of times a fetcher will re-attempt fetching a resource in case of failures.
CrawlerConfig	CrawlerConfig.setFetchersRetryDelay(Duration fetchersRetryDelay)	How long to wait before a failing fetcher re-attempts fetching a resource in case of failures (in milliseconds).
CrawlerConfig	CrawlerConfig.setId(String id)	The crawler unique identifier.
CrawlerConfig	CrawlerConfig.setIdleTimeout(Duration idleTimeout)	The maximum amount of time to wait before shutting down an idle crawler.
CrawlerConfig	CrawlerConfig.setImporterConfig(ImporterConfig importerConfig)	The Importer module configuration.
CrawlerConfig	CrawlerConfig.setMaxCrawlDuration(Duration maxCrawlDuration)	The maximum amount of time a crawler is allowed to run.
CrawlerConfig	CrawlerConfig.setMaxDepth(int maxDepth)	The maximum depth the crawler should go.
CrawlerConfig	CrawlerConfig.setMaxDocuments(int maxDocuments)	The maximum number of documents that can be processed in a single crawler run before stopping.
CrawlerConfig	CrawlerConfig.setMaxQueueBatchSize(int maxQueueBatchSize)	The maximum number of references a node will read at once from the queue, to process locally.
CrawlerConfig	CrawlerConfig.setMaxStreamCachePoolSize(long maxStreamCachePoolSize)	Maximum number of bytes used for memory caching of all reusable streams combined, at any given time, for faster processing.
CrawlerConfig	CrawlerConfig.setMaxStreamCacheSize(long maxStreamCacheSize)	Maximum number of bytes used for memory caching of a single reusable stream, for faster processing.
CrawlerConfig	CrawlerConfig.setMetadataChecksummer(MetadataChecksummer metadataChecksummer)	The metadata checksummer.
CrawlerConfig	CrawlerConfig.setMetadataDeduplicate(boolean metadataDeduplicate)	Whether to turn ON deduplication based on metadata checksum.
CrawlerConfig	CrawlerConfig.setMetadataFetchSupport(FetchDirectiveSupport metadataFetchSupport)
CrawlerConfig	CrawlerConfig.setMetadataFilters(List<MetadataFilter> metadataFilters)	Sets metadata filters.
CrawlerConfig	CrawlerConfig.setMinProgressLoggingInterval(Duration minProgressLoggingInterval)	Minimum amount of time to wait between each logging of crawling progress.
CrawlerConfig	CrawlerConfig.setNumThreads(int numThreads)	The maximum number of threads a crawler can use.
CrawlerConfig	CrawlerConfig.setOrphansStrategy(CrawlerConfig.OrphansStrategy orphansStrategy)	The strategy to adopt when there are orphans.
CrawlerConfig	CrawlerConfig.setPostImportConsumers(List<DocumentConsumer> postImportConsumers)	Sets post-import consumers.
CrawlerConfig	CrawlerConfig.setPreImportConsumers(List<DocumentConsumer> preImportConsumers)	Sets pre-import consumers.
CrawlerConfig	CrawlerConfig.setReferenceFilters(List<ReferenceFilter> referenceFilters)	Sets reference filters.
CrawlerConfig	CrawlerConfig.setSpoiledReferenceStrategizer(SpoiledReferenceStrategizer spoiledReferenceStrategizer)	The spoiled state strategy resolver.
CrawlerConfig	CrawlerConfig.setStartReferences(List<String> startReferences)	Sets the references to initiate crawling from.
CrawlerConfig	CrawlerConfig.setStartReferencesAsync(boolean startReferencesAsync)	Whether the start references should be loaded asynchronously.
CrawlerConfig	CrawlerConfig.setStartReferencesFiles(List<Path> startReferencesFiles)	Sets the file paths of seed files containing references to be used as start references.
CrawlerConfig	CrawlerConfig.setStartReferencesProviders(List<ReferencesProvider> startReferencesProviders)	Sets the providers of references used as starting points for crawling.
CrawlerConfig	CrawlerConfig.setStopOnExceptions(List<Class<? extends Exception>> stopOnExceptions)	Sets the exceptions we want to stop the crawler on.
CrawlerConfig	CrawlerConfig.setWorkDir(Path workDir)	The base directory location where files generated during execution will reside.

Methods in com.norconex.crawler.core that return types with arguments of type CrawlerConfig

Modifier and Type	Method	Description
Class<? extends CrawlerConfig>	CrawlerDriver.crawlerConfigClass()
Consumer<CrawlerConfig>	CrawlerCallbacks.getAfterSession()	Invoked after a crawl session has been destroyed and resources closed.
Consumer<CrawlerConfig>	CrawlerCallbacks.getBeforeSession()	Invoked before the crawl session gets initialized (which is also before any command gets executed).

Method parameters in com.norconex.crawler.core with type arguments of type CrawlerConfig

Modifier and Type	Method	Description
CrawlerCallbacks.CrawlerCallbacksBuilder	CrawlerCallbacks.CrawlerCallbacksBuilder.afterSession(Consumer<CrawlerConfig> afterSession)	Invoked after a crawl session has been destroyed and resources closed.
CrawlerCallbacks.CrawlerCallbacksBuilder	CrawlerCallbacks.CrawlerCallbacksBuilder.beforeSession(Consumer<CrawlerConfig> beforeSession)	Invoked before the crawl session gets initialized (which is also before any command gets executed).
CrawlerDriver.Builder	CrawlerDriver.Builder.crawlerConfigClass(Class<? extends CrawlerConfig> crawlerConfigClass)

Constructors in com.norconex.crawler.core with parameters of type CrawlerConfig

Modifier	Constructor	Description
	Crawler(CrawlerDriver crawlDriver, CrawlerConfig crawlConfig)

Uses of CrawlerConfig in com.norconex.crawler.core.cmd.stop

Constructors in com.norconex.crawler.core.cmd.stop with parameters of type CrawlerConfig

Modifier	Constructor	Description
	StopCommand(CrawlerConfig crawlConfig, String[] nodeUrls)

Uses of CrawlerConfig in com.norconex.crawler.core.context

Methods in com.norconex.crawler.core.context that return CrawlerConfig

Modifier and Type	Method	Description
CrawlerConfig	CrawlerContextFactory.Builder.config()
CrawlerConfig	CrawlerContext.getCrawlConfig()

Methods in com.norconex.crawler.core.context with parameters of type CrawlerConfig

Modifier and Type	Method	Description
CrawlerContextFactory.Builder	CrawlerContextFactory.Builder.config(CrawlerConfig config)
CrawlerContext.Builder	CrawlerContext.Builder.crawlConfig(CrawlerConfig crawlConfig)

Uses of CrawlerConfig in com.norconex.crawler.core.session

Methods in com.norconex.crawler.core.session with parameters of type CrawlerConfig

Modifier and Type	Method	Description
static CrawlerSession	CrawlerSessionFactory.create(CrawlerDriver crawlDriver, CrawlerConfig crawlConfig)

Uses of CrawlerConfig in com.norconex.crawler.core.util

Methods in com.norconex.crawler.core.util with parameters of type CrawlerConfig

Modifier and Type	Method	Description
static String	About.about(CrawlerConfig config)
static void	LogUtil.logCommandIntro(@NonNull Logger logger, @NonNull CrawlerConfig crawlConfig)
static Path	ConfigUtil.resolveTempDir(CrawlerConfig config)
static Path	ConfigUtil.resolveWorkDir(@NonNull CrawlerConfig config)

Uses of CrawlerConfig in com.norconex.crawler.fs

Methods in com.norconex.crawler.fs with parameters of type CrawlerConfig

Modifier and Type	Method	Description
static Crawler	FsCrawler.create(CrawlerConfig crawlConfig)

Uses of CrawlerConfig in com.norconex.crawler.web

Subclasses of CrawlerConfig in com.norconex.crawler.web

Modifier and Type	Class	Description
class	WebCrawlerConfig	Web Crawler configuration, adding more options to the base CrawlerConfig.