Skip to main content

CrawlerConfig

The top-level configuration for a crawler.

crawler/coreJavadoc: CrawlerConfig

Notes​

The top-level crawler configuration. Controls all aspects of a crawl session including start references, fetchers, filters, depth limits, threading, checksumming, deduplication, importer settings, committers, and cluster configuration.

Usage​

changeDiscovery: CRAWLER_SCAN
cluster:
  adminBindAddress: string
  adminDisabled: false
  adminPort: 0
  clustered: false
  connector:
    class: HazelcastClusterConnector
    clusterName: string
    configurer:
      class: JdbcHazelcastConfigurer
    instanceName: string
    nodeExpiryTimeout: 0
    workerHeartbeatInterval: 0
committers: []
deferredShutdownDuration: 0
documentChecksummer:
  class: Md5DocumentChecksummer
  combineFieldsAndContent: false
  fieldMatcher:
    ignoreCase: false
    ignoreDiacritic: false
    matchEmpty: false
    method: BASIC
    negateMatches: false
    partial: false
    pattern: string
    replaceAll: false
    trim: false
  keep: false
  onSet: APPEND
  toField: string
documentDeduplicate: false
documentFetchSupport: DISABLED
documentFilters: []
eventListeners: []
fetchers: []
fetchersMaxRetries: 0
fetchersRetryDelay: 0
id: string
idleTimeout: 0
importer:
  handlers: []
  maxMemoryInstance: 0
  maxMemoryPool: 0
  responseProcessors:
    - {}
  tempDir: string
maxCrawlDuration: 0
maxDepth: 0
maxDocuments: 0
maxQueueBatchSize: 0
maxStreamCachePoolSize: 0
maxStreamCacheSize: 0
metadataChecksummer:
  class: LastModifiedMetadataChecksummer
  keep: false
  onSet: APPEND
  toField: string
metadataDeduplicate: false
metadataFetchSupport: DISABLED
metadataFilters: []
minProgressLoggingInterval: 0
numThreads: 0
orphansStrategy: PROCESS
postImportConsumers: []
preImportConsumers: []
referenceFilters: []
spoiledReferenceStrategizer:
  class: GenericSpoiledReferenceStrategizer
  fallbackStrategy: DELETE
  mappings: {}
startReferences:
  - string
startReferencesAsync: false
startReferencesFiles:
  - string
startReferencesProviders:
  - noop: string
stopOnExceptions:
  - string
workDir: string

Properties​

PropertyTypeRequiredDefault
changeDiscoveryenumNoCRAWLER_SCAN
clusterClusterConfigNoClusterConfig
committersCommitter[]No-
deferredShutdownDurationintegerNo0
documentChecksummerDocumentChecksummerNoMd5DocumentChecksummer
documentDeduplicatebooleanNofalse
documentFetchSupportenumNoREQUIRED
documentFiltersDocumentFilter[]No-
eventListenersEventListener[]No-
fetchersFetcher[]No-
fetchersMaxRetriesintegerNo0
fetchersRetryDelayintegerNo-
idstringYes-
idleTimeoutintegerNo5000
importerImporterConfigNoImporterConfig
maxCrawlDurationintegerNo0
maxDepthintegerNo-1
maxDocumentsintegerNo-1
maxQueueBatchSizeintegerNo50
maxStreamCachePoolSizeintegerNo100000000
maxStreamCacheSizeintegerNo1000000000
metadataChecksummerMetadataChecksummerNo-
metadataDeduplicatebooleanNofalse
metadataFetchSupportenumNoDISABLED
metadataFiltersMetadataFilter[]No-
minProgressLoggingIntervalintegerNo30000
numThreadsintegerNo2
orphansStrategyenumNoPROCESS
postImportConsumersDocumentConsumer[]No-
preImportConsumersDocumentConsumer[]No-
referenceFiltersReferenceFilter[]No-
spoiledReferenceStrategizerSpoiledReferenceStrategizerNoGenericSpoiledReferenceStrategizer
startReferencesstring[]No-
startReferencesAsyncbooleanNofalse
startReferencesFilesstring[]No-
startReferencesProvidersReferencesProvider[]No-
stopOnExceptionsstring[]No-
workDirstringNo-

Property Details​

changeDiscovery​

Defines how the crawler discovers changes during a run. CRAWLER_SCAN (default) discovers changes by scanning source inventory and can treat missing references as orphan evidence in incremental runs. SOURCE_DELTA expects explicit source-side change/delete signals, so missing references alone are not treated as orphan evidence.

Allowed Values​

  • CRAWLER_SCAN
  • SOURCE_DELTA

cluster​

Configuration for the crawler cluster. Controls how the crawler runs across one or more nodes, including distribution of work and inter-node communication.

committers​

One or more Committers responsible for sending crawled documents to their final storage destinations (e.g., search engines, databases). Documents that pass all filters and processing steps are forwarded to each configured Committer.

deferredShutdownDuration​

Amount of time to defer the crawler shutdown after it finishes executing. Useful for giving external processes with polling intervals enough time to capture the latest crawler state before it shuts down. Default is zero (no delay).

documentChecksummer​

Generates a checksum from a document's content to detect whether the document has changed since the last crawl. When combined with documentDeduplicate, duplicate documents within a session are also rejected. Default is MD5-based.

documentDeduplicate​

Whether to enable deduplication based on document checksum. Requires a documentChecksummer to be configured. When enabled, any document with the same checksum as a previously processed document in the same crawl session is rejected. Not recommended unless document checksums are reliably unique for your content.

documentFetchSupport​

Controls whether the crawler performs a full document fetch (e.g., HTTP GET). Can be DISABLED, OPTIONAL (attempts but ignores failure), or REQUIRED (rejects the document on failure). Default is REQUIRED.

Allowed Values​

  • DISABLED
  • OPTIONAL
  • REQUIRED

documentFilters​

Filters applied to documents after their full content has been downloaded. Documents that do not pass the filters are rejected and not sent to the Importer or Committers.

eventListeners​

One or more event listeners that react to crawler lifecycle events (e.g., document fetched, crawl started, crawl completed). Useful for custom monitoring, alerting, or triggering side effects.

fetchers​

One or more Fetchers responsible for obtaining documents and their metadata from the crawl source. At least one fetcher is required. If multiple fetchers are configured, they are tried in order.

fetchersMaxRetries​

Maximum number of times a Fetcher will re-attempt fetching a resource upon failure. Default is zero (no retries).

fetchersRetryDelay​

How long to wait before a failing Fetcher re-attempts fetching a resource, in milliseconds. Default is zero (no delay).

id​

A unique identifier for this crawler. Used to distinguish crawlers within the same crawl session and in generated file names and logs. It is important for this value to be unique among all crawlers in the same session.

idleTimeout​

Maximum time to wait before shutting down an idle crawler (one whose queue is empty and has no active processing). Unlike deferredShutdownDuration, new queue additions can resume processing. Useful when an external process can add references to the queue dynamically. Default is zero (does not wait).

importer​

Configuration for the Norconex Importer module, which handles document content parsing, metadata extraction, and transformation before documents are sent to Committers.

maxCrawlDuration​

Maximum total duration a crawler is allowed to run. Once this limit is reached, the crawler stops processing new documents. Zero or a negative value means unlimited. Default is zero (unlimited).

maxDepth​

Maximum depth the crawler should explore from its start references. The definition of depth is crawler-specific (e.g., directory levels for file system crawlers, click distance from start URL for web crawlers). Default is -1 (unlimited).

maxDocuments​

Maximum number of documents processed in a single crawler run before the crawler stops. If the queue still contains documents when this limit is reached, the crawl session is paused, not completed, and the next run resumes the same session.

maxQueueBatchSize​

Maximum number of references a crawler node reads at once from the queue to process locally. Adjusting this value can affect throughput in clustered deployments. Default is 50.

maxStreamCachePoolSize​

Maximum number of bytes used for in-memory caching of all reusable streams combined at any given time. Defaults to 1 GB. File-based caching is used when this limit is reached.

maxStreamCacheSize​

Maximum number of bytes used for in-memory caching of a single reusable stream. Defaults to 100 MB. File-based caching is used when this limit is reached for a single stream, or when the pool maximum has been reached.

metadataChecksummer​

Generates a checksum from a document's metadata to detect whether the metadata has changed since the last crawl. When combined with metadataDeduplicate, duplicate metadata within a session are also rejected. Metadata checksum generation is disabled when not configured.

metadataDeduplicate​

Whether to enable deduplication based on metadata checksum. Requires a metadataChecksummer to be configured. When enabled, any document with the same metadata checksum as a previously processed document in the same crawl session is rejected. Not recommended unless metadata checksums are reliably unique.

metadataFetchSupport​

Controls whether the crawler fetches document metadata separately before (or instead of) fetching the full document. Can be DISABLED, OPTIONAL, or REQUIRED. Default is DISABLED.

Allowed Values​

  • DISABLED
  • OPTIONAL
  • REQUIRED

metadataFilters​

Filters applied to documents based on their metadata, after metadata has been obtained but before the full document is fetched. Documents that do not pass the filters are rejected.

minProgressLoggingInterval​

Minimum time to wait between each crawling progress log entry. A value of null or below 1 second disables progress logging. Default is 30 seconds.

numThreads​

Maximum number of concurrent threads the crawler uses for processing documents. Default is 2. In clustered deployments, this value applies per node.

orphansStrategy​

Determines what to do with previously crawled documents that are no longer found during the current crawl (orphans). Typical values are PROCESS, DELETE, or IGNORE.

Allowed Values​

  • PROCESS
  • DELETE
  • IGNORE

postImportConsumers​

A list of document consumers invoked after the document has been processed by the importer.

preImportConsumers​

A list of document consumers invoked before the document is sent to the importer for content extraction and metadata enrichment.

referenceFilters​

Filters applied to document references (URLs or file paths) to control which ones are fetched. Documents whose references do not pass the filters are skipped.

spoiledReferenceStrategizer​

Determines how to handle references that repeatedly fail to be fetched or processed. Configures the strategy for dealing with such "spoiled" references.

startReferences​

The initial set of document references (URLs or file paths) from which the crawl begins.

startReferencesAsync​

When enabled, start references are provided and consumed asynchronously, allowing the crawl to begin before all start references are available.

startReferencesFiles​

One or more files containing lists of start references, one reference per line.

startReferencesProviders​

A list of pluggable providers that supply the initial references to crawl, used when start references come from a dynamic or external source.

stopOnExceptions​

A list of fully qualified exception class names which, when encountered, will stop the crawler.

workDir​

The directory where working files (temporary data, queues, state) are stored during crawl execution.