Package com.norconex.crawler.core

Class CrawlerConfig

java.lang.Object
com.norconex.crawler.core.CrawlerConfig
Direct Known Subclasses:
WebCrawlerConfig
public class CrawlerConfig extends Object

Base Crawler configuration. Crawlers usually read this configuration upon starting up. While not always enforced, once execution has started, it should be considered immutable to avoid unexpected behaviors.

  • Field Details

    • DEFAULT_WORKDIR

      public static final Path DEFAULT_WORKDIR
      Default relative directory where generated files are written.

    • DEFAULT_IDLE_TIMEOUT

      public static final Duration DEFAULT_IDLE_TIMEOUT

    • DEFAULT_MIN_PROGRESS_LOGGING_INTERVAL

      public static final Duration DEFAULT_MIN_PROGRESS_LOGGING_INTERVAL

  • Constructor Details

    • CrawlerConfig

      public CrawlerConfig()

  • Method Details

    • getStartReferences

      public List<String> getStartReferences()
      Gets the references to initiate crawling from.
      Returns:
      start references (never null)

    • setStartReferences

      public CrawlerConfig setStartReferences(List<String> startReferences)
      Sets the references to initiate crawling from.
      Parameters:
      startReferences - start references
      Returns:
      this

    • getStartReferencesFiles

      public List<Path> getStartReferencesFiles()
      Gets the file paths of seed files containing references to be used as start references. Files are expected to have one reference per line. Blank lines and lines starting with # (comment) are ignored.
      Returns:
      file paths of seed files containing references (never null)

    • setStartReferencesFiles

      public CrawlerConfig setStartReferencesFiles(List<Path> startReferencesFiles)
      Sets the file paths of seed files containing references to be used as start references. Files are expected to have one reference per line. Blank lines and lines starting with # (comment) are ignored.
      Parameters:
      startReferencesFiles - file paths of seed files containing references
      Returns:
      this

    • getStartReferencesProviders

      public List<ReferencesProvider> getStartReferencesProviders()
      Gets the providers of references used as starting points for crawling. Use this approach when references need to be provided dynamically at launch time.
      Returns:
      start references providers (never null)

    • setStartReferencesProviders

      public CrawlerConfig setStartReferencesProviders(List<ReferencesProvider> startReferencesProviders)
      Sets the providers of references used as starting points for crawling. Use this approach when references need to be provided dynamically at launch time.
      Parameters:
      startReferencesProviders - start references provider
      Returns:
      this

    • getStopOnExceptions

      public List<Class<? extends Exception>> getStopOnExceptions()
      The exceptions we want to stop the crawler on. By default the crawler will log exceptions from processing a document and try to move on to the next without stopping. Even if no exceptions are returned by this method, the crawler can sometimes stop regardless if it cannot recover safely from an exception. To capture more exceptions, use a parent class (e.g., Exception should catch them all).
      Returns:
      exceptions that will stop the crawler when encountered

    • setStopOnExceptions

      public CrawlerConfig setStopOnExceptions(List<Class<? extends Exception>> stopOnExceptions)
      Sets the exceptions we want to stop the crawler on. By default the crawler will log exceptions from processing a document and try to move on to the next without stopping. Even if no exceptions are returned by this method, the crawler can sometimes stop regardless if it cannot recover safely from an exception. To capture more exceptions, use a parent class (e.g., Exception should catch them all).
      Parameters:
      stopOnExceptions - exceptions that will stop the crawler when encountered
      Returns:
      this

    • getReferenceFilters

      public List<ReferenceFilter> getReferenceFilters()
      Gets reference filters
      Returns:
      reference filters

    • setReferenceFilters

      public CrawlerConfig setReferenceFilters(List<ReferenceFilter> referenceFilters)
      Sets reference filters.
      Parameters:
      referenceFilters - the referenceFilters to set
      Returns:
      this

    • getDocumentFilters

      public List<DocumentFilter> getDocumentFilters()
      Gets the document filters.
      Returns:
      document filters

    • setDocumentFilters

      public CrawlerConfig setDocumentFilters(List<DocumentFilter> documentFilters)
      Sets document filters.
      Parameters:
      documentFilters - document filters
      Returns:
      this

    • getMetadataFilters

      public List<MetadataFilter> getMetadataFilters()
      Gets metadata filters.
      Returns:
      metadata filters

    • setMetadataFilters

      public CrawlerConfig setMetadataFilters(List<MetadataFilter> metadataFilters)
      Sets metadata filters.
      Parameters:
      metadataFilters - metadata filters
      Returns:
      this

    • getCommitters

      public List<Committer> getCommitters()
      Gets Committers responsible for persisting information to a target location/repository.
      Returns:
      list of Committers (never null)

    • setCommitters

      public CrawlerConfig setCommitters(List<Committer> committers)
      Sets Committers responsible for persisting information to a target location/repository.
      Parameters:
      committers - list of Committers
      Returns:
      this

    • addCommitter

      public CrawlerConfig addCommitter(Committer committer)
      Adds a Committer responsible for persisting information to a target location/repository.
      Parameters:
      committer - a Committer
      Returns:
      this

    • getEventListeners

      public List<EventListener<?>> getEventListeners()
      Gets event listeners. Those are considered additions to automatically detected configuration objects implementing EventListener.
      Returns:
      event listeners.

    • setEventListeners

      public CrawlerConfig setEventListeners(List<EventListener<?>> eventListeners)
      Sets event listeners. Those are considered additions to automatically detected configuration objects implementing EventListener.
      Parameters:
      eventListeners - event listeners.
      Returns:
      this

    • addEventListeners

      public CrawlerConfig addEventListeners(List<EventListener<?>> eventListeners)
      Adds event listeners. Those are considered additions to automatically detected configuration objects implementing EventListener.
      Parameters:
      eventListeners - event listeners.
      Returns:
      this

    • addEventListener

      public CrawlerConfig addEventListener(EventListener<?> eventListener)
      Adds an event listener. Those are considered additions to automatically detected configuration objects implementing EventListener.
      Parameters:
      eventListener - event listener.
      Returns:
      this

    • removeEventListener

      public boolean removeEventListener(EventListener<?> eventListener)
      Removes a single event listener from the list of registered listeners if present.
      Parameters:
      eventListener - event listener
      Returns:
      true if the entity listener existed

    • clearEventListeners

      public CrawlerConfig clearEventListeners()
      Clears all event listeners. The automatically detected configuration objects implementing EventListener are not cleared.
      Returns:
      this

    • getPreImportConsumers

      public List<DocumentConsumer> getPreImportConsumers()
      Gets pre-import consumers.
      Returns:
      pre-import consumers

    • setPreImportConsumers

      public CrawlerConfig setPreImportConsumers(List<DocumentConsumer> preImportConsumers)
      Sets pre-import consumers.
      Parameters:
      preImportConsumers - pre-import consumers
      Returns:
      this

    • getPostImportConsumers

      public List<DocumentConsumer> getPostImportConsumers()
      Gets post-import consumers.
      Returns:
      post-import consumers

    • setPostImportConsumers

      public CrawlerConfig setPostImportConsumers(List<DocumentConsumer> postImportConsumers)
      Sets post-import consumers.
      Parameters:
      postImportConsumers - post-import consumers
      Returns:
      this

    • getFetchers

      public List<Fetcher> getFetchers()
      One or more fetchers responsible for pulling documents and document metadata associated with a reference from a source. When more than one are configured and for each documents, fetchers will be invoked in their defined order, until the first one that accepts and successfully process a reference (others are not invoked).
      Returns:
      one or more fetchers

    • setFetchers

      public CrawlerConfig setFetchers(List<Fetcher> fetchers)
      One or more fetchers responsible for pulling documents and document metadata associated with a reference from a source. When more than one are configured and for each documents, fetchers will be invoked in their defined order, until the first one that accepts and successfully process a reference (others are not invoked).
      Parameters:
      fetchers - one or more fetchers
      Returns:
      this

    • getId

      public String getId()
      The crawler unique identifier. Using usual names is perfectly fine (non-alphanumeric characters are OK). It is important for this crawler ID to be unique amongst your crawlers in the same crawl session. On top of avoiding conflicts, it facilitates integration with different systems and facilitates tracking.

    • getWorkDir

      public Path getWorkDir()
      The base directory location where files generated during execution will reside. When null the collector will use ./work, relative to the execution "current" directory.

    • getMaxStreamCachePoolSize

      public long getMaxStreamCachePoolSize()
      Maximum number of bytes used for memory caching of all reusable streams combined, at any given time, for faster processing. Defaults to 1 GB. File-caching is used when the maximum is reached.

    • getMaxStreamCacheSize

      public long getMaxStreamCacheSize()
      Maximum number of bytes used for memory caching of a single reusable stream, for faster processing. Defaults to 100 MB. File-caching is used when this maximum is reached for a single file, or when the pool maximum size has been reached.

    • getDeferredShutdownDuration

      public Duration getDeferredShutdownDuration()
      The amount of time to defer the crawler shutdown when it is done executing. This is useful for giving external processes with polling intervals enough time to grab the latest state of the collector before it shuts down. Default is zero (does not wait to shutdown after completion).

    • getMaxCrawlDuration

      public Duration getMaxCrawlDuration()
      The maximum amount of time a crawler is allowed to run. Zero or a negative value means unlimited. Default is zero (unlimited).

    • getClusterConfig

      @NonNull public @NonNull ClusterConfig getClusterConfig()
      The Importer module configuration.

    • isStartReferencesAsync

      public boolean isStartReferencesAsync()
      Whether the start references should be loaded asynchronously. When true, the crawler will start processing the start references in one or more separate threads as they are added to the queue (as opposed to wait for queue initialization to be complete). While this may speed up crawling, it may have an unexpected effect on accuracy of CrawlerDocMetaConstants.DEPTH. Use of this option is only recommended when start references take a significant time to load.

    • getNumThreads

      public int getNumThreads()
      The maximum number of threads a crawler can use. Default is 2.

      When running in a cluster, this value applies per node. For example, setting numThreads to 4 on a 3-node cluster results in up to 12 concurrent processing threads across the cluster.

    • getMaxDocuments

      public int getMaxDocuments()

      The maximum number of documents that can be processed in a single crawler run before stopping. Not all processed documents make it to your Committers as some can be rejected.

      In multi-threaded or clustered environments, the actual number of documents processed may be a bit higher than the specified maximum due to concurrency. Upon reaching the configured maximum, the crawler will finish with its documents actively being processed before stopping.

      Reaching the maximum value stops the current run, but it does not otherwise consider the crawler session "complete". If there are still queued documents left to process, the session is effectively paused and the next run resumes it, processing an additional number of documents up to the same maximum. This maximum therefore allows crawling one or more sources in chunks, processing a bounded number of documents on each run. When the session fully completes, the next run starts a new crawl session. To prevent resuming a partial crawl session, explicitly clean the crawl store first.

      For more control on what events may stop the crawler, consider using configuring a StopCrawlerOnMaxEventListener.

      Default is -1 (unlimited).

    • getMaxQueueBatchSize

      public int getMaxQueueBatchSize()
      The maximum number of references a node will read at once from the queue, to process locally.

    • getMaxDepth

      public int getMaxDepth()
      The maximum depth the crawler should go. The exact definition of depth is crawler-specific. Examples: levels of sub-directories, number of URL clicks to reach a page, etc. Refer to specific crawler implementation for details. Default is -1 (unlimited).

    • getIdleTimeout

      public Duration getIdleTimeout()
      The maximum amount of time to wait before shutting down an idle crawler. A crawler is considered idle when its queue is empty and there are no reference being actively processed. Differs from deferredShutdownDuration in that additions to the crawler queue will resume the normal processing. A non-zero value can be useful if the crawler queue can be populated by an external process. Defaults to DEFAULT_IDLE_TIMEOUT (5 seconds). A null or zero value disables the wait. The smallest considered unit is seconds (milliseconds are rounded up).

    • getMinProgressLoggingInterval

      public Duration getMinProgressLoggingInterval()
      Minimum amount of time to wait between each logging of crawling progress. Minimum value is 1 second. Defaults to DEFAULT_MIN_PROGRESS_LOGGING_INTERVAL (30 seconds). A null value or a value below 1 second disables progress logging.

    • getOrphansStrategy

      public CrawlerConfig.OrphansStrategy getOrphansStrategy()

      The strategy to adopt when there are orphans. Orphans are references that were processed in a previous run, but were not in the current run. In other words, they are leftovers from a previous run that were not re-encountered in the current.

      Unless explicitly stated otherwise by a crawler implementation, the default strategy is to PROCESS orphans. Setting a null value is the same as setting IGNORE.

      Be careful: Setting the orphan strategy to DELETE is NOT recommended in most cases. There are times when a temporary failure such as a network outage or a web page timing out, may cause some documents not to be crawled. When this happens, unreachable documents would be considered "orphans" and be deleted while under normal circumstances, they should be kept. Re-processing them (default), is usually the safest approach to confirm they still exist before deleting or updating them.

    • getChangeDiscovery

      public CrawlerConfig.ChangeDiscovery getChangeDiscovery()
      Change discovery strategy. Defaults to crawler-driven inventory scan.

    • getMetadataChecksummer

      public MetadataChecksummer getMetadataChecksummer()
      The metadata checksummer. Metadata checksum generation is disabled when null.

    • getImporterConfig

      public ImporterConfig getImporterConfig()
      The Importer module configuration.

    • isMetadataDeduplicate

      public boolean isMetadataDeduplicate()
      Whether to turn ON deduplication based on metadata checksum. To enable, getMetadataChecksummer() must not return null. Not recommended unless you know for sure your metadata checksum is acceptably unique.

    • isDocumentDeduplicate

      public boolean isDocumentDeduplicate()
      Whether to turn ON deduplication based on document checksum. To enable, getDocumentChecksummer() must not return null. Not recommended unless you know for sure your document checksum is acceptably unique.

    • getDocumentChecksummer

      public DocumentChecksummer getDocumentChecksummer()
      The document checksummer. Document checksum generation is disabled when null.

    • getSpoiledReferenceStrategizer

      public SpoiledReferenceStrategizer getSpoiledReferenceStrategizer()
      The spoiled state strategy resolver. A spoiled document is one that was crawled properly before but on a subsequent crawl, it can no longer be crawled for whatever reason (not found, bad status, server error, etc.).

    • getMetadataFetchSupport

      public FetchDirectiveSupport getMetadataFetchSupport()

    • getDocumentFetchSupport

      public FetchDirectiveSupport getDocumentFetchSupport()

    • getFetchersMaxRetries

      public int getFetchersMaxRetries()
      The maximum number of times a fetcher will re-attempt fetching a resource in case of failures. Default is zero (won't retry).

    • getFetchersRetryDelay

      public Duration getFetchersRetryDelay()
      How long to wait before a failing fetcher re-attempts fetching a resource in case of failures (in milliseconds). Default is zero (no delay).

    • setId

      public CrawlerConfig setId(String id)
      The crawler unique identifier. Using usual names is perfectly fine (non-alphanumeric characters are OK). It is important for this crawler ID to be unique amongst your crawlers in the same crawl session. On top of avoiding conflicts, it facilitates integration with different systems and facilitates tracking.
      Returns:
      this.

    • setWorkDir

      public CrawlerConfig setWorkDir(Path workDir)
      The base directory location where files generated during execution will reside. When null the collector will use ./work, relative to the execution "current" directory.
      Returns:
      this.

    • setMaxStreamCachePoolSize

      public CrawlerConfig setMaxStreamCachePoolSize(long maxStreamCachePoolSize)
      Maximum number of bytes used for memory caching of all reusable streams combined, at any given time, for faster processing. Defaults to 1 GB. File-caching is used when the maximum is reached.
      Returns:
      this.

    • setMaxStreamCacheSize

      public CrawlerConfig setMaxStreamCacheSize(long maxStreamCacheSize)
      Maximum number of bytes used for memory caching of a single reusable stream, for faster processing. Defaults to 100 MB. File-caching is used when this maximum is reached for a single file, or when the pool maximum size has been reached.
      Returns:
      this.

    • setDeferredShutdownDuration

      public CrawlerConfig setDeferredShutdownDuration(Duration deferredShutdownDuration)
      The amount of time to defer the crawler shutdown when it is done executing. This is useful for giving external processes with polling intervals enough time to grab the latest state of the collector before it shuts down. Default is zero (does not wait to shutdown after completion).
      Returns:
      this.

    • setMaxCrawlDuration

      public CrawlerConfig setMaxCrawlDuration(Duration maxCrawlDuration)
      The maximum amount of time a crawler is allowed to run. Zero or a negative value means unlimited. Default is zero (unlimited).
      Returns:
      this.

    • setClusterConfig

      public CrawlerConfig setClusterConfig(@NonNull @NonNull ClusterConfig clusterConfig)
      The Importer module configuration.
      Returns:
      this.

    • setStartReferencesAsync

      public CrawlerConfig setStartReferencesAsync(boolean startReferencesAsync)
      Whether the start references should be loaded asynchronously. When true, the crawler will start processing the start references in one or more separate threads as they are added to the queue (as opposed to wait for queue initialization to be complete). While this may speed up crawling, it may have an unexpected effect on accuracy of CrawlerDocMetaConstants.DEPTH. Use of this option is only recommended when start references take a significant time to load.
      Returns:
      this.

    • setNumThreads

      public CrawlerConfig setNumThreads(int numThreads)
      The maximum number of threads a crawler can use. Default is 2.

      When running in a cluster, this value applies per node. For example, setting numThreads to 4 on a 3-node cluster results in up to 12 concurrent processing threads across the cluster.

      Returns:
      this.

    • setMaxDocuments

      public CrawlerConfig setMaxDocuments(int maxDocuments)

      The maximum number of documents that can be processed in a single crawler run before stopping. Not all processed documents make it to your Committers as some can be rejected.

      In multi-threaded or clustered environments, the actual number of documents processed may be a bit higher than the specified maximum due to concurrency. Upon reaching the configured maximum, the crawler will finish with its documents actively being processed before stopping.

      Reaching the maximum value stops the current run, but it does not otherwise consider the crawler session "complete". If there are still queued documents left to process, the session is effectively paused and the next run resumes it, processing an additional number of documents up to the same maximum. This maximum therefore allows crawling one or more sources in chunks, processing a bounded number of documents on each run. When the session fully completes, the next run starts a new crawl session. To prevent resuming a partial crawl session, explicitly clean the crawl store first.

      For more control on what events may stop the crawler, consider using configuring a StopCrawlerOnMaxEventListener.

      Default is -1 (unlimited).

      Returns:
      this.

    • setMaxQueueBatchSize

      public CrawlerConfig setMaxQueueBatchSize(int maxQueueBatchSize)
      The maximum number of references a node will read at once from the queue, to process locally.
      Returns:
      this.

    • setMaxDepth

      public CrawlerConfig setMaxDepth(int maxDepth)
      The maximum depth the crawler should go. The exact definition of depth is crawler-specific. Examples: levels of sub-directories, number of URL clicks to reach a page, etc. Refer to specific crawler implementation for details. Default is -1 (unlimited).
      Returns:
      this.

    • setIdleTimeout

      public CrawlerConfig setIdleTimeout(Duration idleTimeout)
      The maximum amount of time to wait before shutting down an idle crawler. A crawler is considered idle when its queue is empty and there are no reference being actively processed. Differs from deferredShutdownDuration in that additions to the crawler queue will resume the normal processing. A non-zero value can be useful if the crawler queue can be populated by an external process. Defaults to DEFAULT_IDLE_TIMEOUT (5 seconds). A null or zero value disables the wait. The smallest considered unit is seconds (milliseconds are rounded up).
      Returns:
      this.

    • setMinProgressLoggingInterval

      public CrawlerConfig setMinProgressLoggingInterval(Duration minProgressLoggingInterval)
      Minimum amount of time to wait between each logging of crawling progress. Minimum value is 1 second. Defaults to DEFAULT_MIN_PROGRESS_LOGGING_INTERVAL (30 seconds). A null value or a value below 1 second disables progress logging.
      Returns:
      this.

    • setOrphansStrategy

      public CrawlerConfig setOrphansStrategy(CrawlerConfig.OrphansStrategy orphansStrategy)

      The strategy to adopt when there are orphans. Orphans are references that were processed in a previous run, but were not in the current run. In other words, they are leftovers from a previous run that were not re-encountered in the current.

      Unless explicitly stated otherwise by a crawler implementation, the default strategy is to PROCESS orphans. Setting a null value is the same as setting IGNORE.

      Be careful: Setting the orphan strategy to DELETE is NOT recommended in most cases. There are times when a temporary failure such as a network outage or a web page timing out, may cause some documents not to be crawled. When this happens, unreachable documents would be considered "orphans" and be deleted while under normal circumstances, they should be kept. Re-processing them (default), is usually the safest approach to confirm they still exist before deleting or updating them.

      Returns:
      this.

    • setChangeDiscovery

      public CrawlerConfig setChangeDiscovery(CrawlerConfig.ChangeDiscovery changeDiscovery)
      Change discovery strategy. Defaults to crawler-driven inventory scan.
      Returns:
      this.

    • setMetadataChecksummer

      public CrawlerConfig setMetadataChecksummer(MetadataChecksummer metadataChecksummer)
      The metadata checksummer. Metadata checksum generation is disabled when null.
      Returns:
      this.

    • setImporterConfig

      public CrawlerConfig setImporterConfig(ImporterConfig importerConfig)
      The Importer module configuration.
      Returns:
      this.

    • setMetadataDeduplicate

      public CrawlerConfig setMetadataDeduplicate(boolean metadataDeduplicate)
      Whether to turn ON deduplication based on metadata checksum. To enable, getMetadataChecksummer() must not return null. Not recommended unless you know for sure your metadata checksum is acceptably unique.
      Returns:
      this.

    • setDocumentDeduplicate

      public CrawlerConfig setDocumentDeduplicate(boolean documentDeduplicate)
      Whether to turn ON deduplication based on document checksum. To enable, getDocumentChecksummer() must not return null. Not recommended unless you know for sure your document checksum is acceptably unique.
      Returns:
      this.

    • setDocumentChecksummer

      public CrawlerConfig setDocumentChecksummer(DocumentChecksummer documentChecksummer)
      The document checksummer. Document checksum generation is disabled when null.
      Returns:
      this.

    • setSpoiledReferenceStrategizer

      public CrawlerConfig setSpoiledReferenceStrategizer(SpoiledReferenceStrategizer spoiledReferenceStrategizer)
      The spoiled state strategy resolver. A spoiled document is one that was crawled properly before but on a subsequent crawl, it can no longer be crawled for whatever reason (not found, bad status, server error, etc.).
      Returns:
      this.

    • setMetadataFetchSupport

      public CrawlerConfig setMetadataFetchSupport(FetchDirectiveSupport metadataFetchSupport)
      Returns:
      this.

    • setDocumentFetchSupport

      public CrawlerConfig setDocumentFetchSupport(FetchDirectiveSupport documentFetchSupport)
      Returns:
      this.

    • setFetchersMaxRetries

      public CrawlerConfig setFetchersMaxRetries(int fetchersMaxRetries)
      The maximum number of times a fetcher will re-attempt fetching a resource in case of failures. Default is zero (won't retry).
      Returns:
      this.

    • setFetchersRetryDelay

      public CrawlerConfig setFetchersRetryDelay(Duration fetchersRetryDelay)
      How long to wait before a failing fetcher re-attempts fetching a resource in case of failures (in milliseconds). Default is zero (no delay).
      Returns:
      this.

    • equals

      public boolean equals(Object o)
      Overrides:
      equals in class Object

    • canEqual

      protected boolean canEqual(Object other)

    • hashCode

      public int hashCode()
      Overrides:
      hashCode in class Object

    • toString

      public String toString()
      Overrides:
      toString in class Object