Skip to main content

WebCrawlerConfig

The top-level configuration for a web crawler.

crawler/webJavadoc: WebCrawlerConfig

Notes​

The top-level configuration for a web crawler. Extends the base crawl configuration with web-specific settings including link extractors, URL normalization, robots.txt and robots meta handling, sitemap support, URL scope resolution, canonical link detection, delay resolution, and recrawl scheduling.

Usage​

canonicalLinkDetector:
  class: GenericCanonicalLinkDetector
  contentTypes:
    - string
changeDiscovery: CRAWLER_SCAN
cluster:
  adminBindAddress: string
  adminDisabled: false
  adminPort: 0
  clustered: false
  connector:
    class: HazelcastClusterConnector
    clusterName: string
    configurer:
      class: JdbcHazelcastConfigurer
    instanceName: string
    nodeExpiryTimeout: 0
    workerHeartbeatInterval: 0
committers: []
deferredShutdownDuration: 0
delayResolver:
  class: GenericDelayResolver
  defaultDelay: 0
  ignoreRobotsCrawlDelay: false
  schedules:
    - {}
  scope: CRAWLER
documentChecksummer:
  class: Md5DocumentChecksummer
  combineFieldsAndContent: false
  fieldMatcher:
    ignoreCase: false
    ignoreDiacritic: false
    matchEmpty: false
    method: BASIC
    negateMatches: false
    partial: false
    pattern: string
    replaceAll: false
    trim: false
  keep: false
  onSet: APPEND
  toField: string
documentDeduplicate: false
documentFetchSupport: DISABLED
documentFilters: []
eventListeners: []
fetchers: []
fetchersMaxRetries: 0
fetchersRetryDelay: 0
id: string
idleTimeout: 0
importer:
  handlers: []
  maxMemoryInstance: 0
  maxMemoryPool: 0
  responseProcessors:
    - {}
  tempDir: string
keepReferencedLinks:
  - INSCOPE
linkExtractors: []
maxCrawlDuration: 0
maxDepth: 0
maxDocuments: 0
maxQueueBatchSize: 0
maxStreamCachePoolSize: 0
maxStreamCacheSize: 0
metadataChecksummer:
  class: LastModifiedMetadataChecksummer
  keep: false
  onSet: APPEND
  toField: string
metadataDeduplicate: false
metadataFetchSupport: DISABLED
metadataFilters: []
minProgressLoggingInterval: 0
numThreads: 0
orphansStrategy: PROCESS
postImportConsumers: []
postImportLinks:
  ignoreCase: false
  ignoreDiacritic: false
  matchEmpty: false
  method: BASIC
  negateMatches: false
  partial: false
  pattern: string
  replaceAll: false
  trim: false
postImportLinksKeep: false
preImportConsumers: []
recrawlableResolver:
  class: GenericRecrawlableResolver
  minFrequencies:
    - {}
  sitemapSupport: FIRST
referenceFilters: []
robotsMetaProvider:
  class: StandardRobotsMetaProvider
  headersPrefix: string
robotsTxtProvider:
  class: StandardRobotsTxtProvider
  noop: string
sitemapLocator:
  class: GenericSitemapLocator
  paths:
    - string
  robotsTxtSitemapDisabled: false
sitemapResolver:
  class: GenericSitemapResolver
  lenient: false
spoiledReferenceStrategizer:
  class: GenericSpoiledReferenceStrategizer
  fallbackStrategy: DELETE
  mappings: {}
startReferences:
  - string
startReferencesAsync: false
startReferencesFiles:
  - string
startReferencesProviders:
  - noop: string
startReferencesSitemaps:
  - string
stopOnExceptions:
  - string
urlNormalizers: []
urlScopeResolver:
  class: GenericUrlScopeResolver
  includeSubdomains: false
  stayOnDomain: false
  stayOnPort: false
  stayOnProtocol: false
  stayOnSitemap: false
workDir: string

Properties​

PropertyTypeRequiredDefault
canonicalLinkDetectorCanonicalLinkDetectorNoGenericCanonicalLinkDetector
changeDiscoveryenumNoCRAWLER_SCAN
clusterClusterConfigNoClusterConfig
committersCommitter[]No-
deferredShutdownDurationintegerNo0
delayResolverDelayResolverNoGenericDelayResolver
documentChecksummerDocumentChecksummerNoMd5DocumentChecksummer
documentDeduplicatebooleanNofalse
documentFetchSupportenumNoREQUIRED
documentFiltersDocumentFilter[]No-
eventListenersEventListener[]No-
fetchersFetcher[]No[HttpClientFetcher]
fetchersMaxRetriesintegerNo0
fetchersRetryDelayintegerNo-
idstringYes-
idleTimeoutintegerNo5000
importerImporterConfigNoImporterConfig
linkExtractorsLinkExtractor[]No[HtmlLinkExtractor]
maxCrawlDurationintegerNo0
maxDepthintegerNo-1
maxDocumentsintegerNo-1
maxQueueBatchSizeintegerNo50
maxStreamCachePoolSizeintegerNo100000000
maxStreamCacheSizeintegerNo1000000000
metadataChecksummerMetadataChecksummerNoLastModifiedMetadataChecksummer
metadataDeduplicatebooleanNofalse
metadataFetchSupportenumNoDISABLED
metadataFiltersMetadataFilter[]No-
minProgressLoggingIntervalintegerNo30000
numThreadsintegerNo2
orphansStrategyenumNoPROCESS
postImportConsumersDocumentConsumer[]No-
postImportLinksKeepbooleanNofalse
preImportConsumersDocumentConsumer[]No-
recrawlableResolverRecrawlableResolverNoGenericRecrawlableResolver
referenceFiltersReferenceFilter[]No-
robotsMetaProviderRobotsMetaProviderNoStandardRobotsMetaProvider
robotsTxtProviderRobotsTxtProviderNoStandardRobotsTxtProvider
sitemapLocatorSitemapLocatorNoGenericSitemapLocator
sitemapResolverSitemapResolverNoGenericSitemapResolver
spoiledReferenceStrategizerSpoiledReferenceStrategizerNoGenericSpoiledReferenceStrategizer
startReferencesstring[]No-
startReferencesAsyncbooleanNofalse
startReferencesFilesstring[]No-
startReferencesProvidersReferencesProvider[]No-
startReferencesSitemapsstring[]No-
stopOnExceptionsstring[]No-
urlNormalizersWebUrlNormalizer[]No[GenericUrlNormalizer]
urlScopeResolverUrlScopeResolverNoGenericUrlScopeResolver
workDirstringNo-

Property Details​

canonicalLinkDetector​

The detector used to identify canonical link declarations in HTTP responses and HTML meta tags, rejecting non-canonical URLs to avoid duplicates. Defaults to GenericCanonicalLinkDetector. Set to null to disable canonical link detection.

changeDiscovery​

Defines how the crawler discovers changes during a run. CRAWLER_SCAN (default) discovers changes by scanning source inventory and can treat missing references as orphan evidence in incremental runs. SOURCE_DELTA expects explicit source-side change/delete signals, so missing references alone are not treated as orphan evidence.

Allowed Values​

  • CRAWLER_SCAN
  • SOURCE_DELTA

cluster​

Cluster configuration for the crawler, including the connector type, whether clustering is enabled, and administrative port settings.

committers​

One or more Committers responsible for persisting crawled documents to a target repository. Multiple committers can be configured to send data to different destinations simultaneously.

deferredShutdownDuration​

The amount of time to defer crawler shutdown after it finishes executing. Useful for giving external processes with polling intervals enough time to read the final state before the crawler exits. Default is zero (no deferred shutdown).

delayResolver​

Dictates the minimum amount of time to wait between web requests, helping crawlers be respectful of target web sites. Defaults to GenericDelayResolver.

documentChecksummer​

Generates a checksum from a document's content to detect changes between crawl sessions. Set to null to disable document checksumming. Defaults to Md5DocumentChecksummer.

documentDeduplicate​

When enabled, documents sharing the same document checksum within a crawl session are treated as duplicates and rejected. Requires a document checksummer to be configured. Not recommended unless checksums are guaranteed to be sufficiently unique.

documentFetchSupport​

Controls how the HTTP GET request (document fetch) is handled. Supported values are DISABLED (no GET request is made), OPTIONAL (a failed GET does not stop processing if HEAD succeeded), and REQUIRED (a failed GET rejects the document). Default is REQUIRED.

Allowed Values​

  • DISABLED
  • OPTIONAL
  • REQUIRED

documentFilters​

Filters applied after a document is downloaded but before it is imported, allowing rejection based on the document's content or metadata.

eventListeners​

A list of event listeners that are notified of crawler events such as document rejection, import completion, commit, and more. These are in addition to any configuration objects that automatically implement the listener interface.

fetchers​

One or more fetchers responsible for downloading documents and their metadata. Fetchers are tried in order; the first one that accepts and successfully processes a reference is used, and the others are skipped. Defaults to HttpClientFetcher.

fetchersMaxRetries​

The maximum number of times a fetcher will re-attempt fetching a resource after a failure. Default is zero (no retries).

fetchersRetryDelay​

The amount of time to wait before a failing fetcher re-attempts fetching a resource. Default is zero (no delay between retries).

id​

The unique identifier for this crawler. Must be unique among all crawlers in the same crawl session. Used to distinguish crawlers and facilitate integration and tracking.

idleTimeout​

The maximum amount of time to wait before shutting down a crawler that has an empty queue and no active processing. A non-zero value is useful when an external process populates the queue asynchronously. Default is 5 seconds.

importer​

The Importer module configuration, responsible for transforming, parsing, and enriching documents after they are fetched.

Specifies which types of extracted links are stored as document metadata. Supported values are INSCOPE (default, stores in-scope links), OUTSCOPE (stores out-of-scope links), and MAXDEPTH (also stores links found on pages at the maximum crawl depth).

linkExtractors​

One or more link extractors responsible for identifying URLs in crawled documents for further crawling. Multiple extractors can handle different content types. Defaults to HtmlLinkExtractor.

maxCrawlDuration​

The maximum amount of time a crawler is allowed to run. A zero or negative value means unlimited. Default is zero (unlimited).

maxDepth​

The maximum crawl depth, representing how many link-levels away from a start URL the crawler will follow. Default is -1 (unlimited).

maxDocuments​

Maximum number of documents processed in a single crawler run before the crawler stops. If the queue still contains documents when this limit is reached, the crawl session is paused, not completed, and the next run resumes the same session.

maxQueueBatchSize​

The maximum number of references a node reads at once from the queue to process locally. Default is 50.

maxStreamCachePoolSize​

The maximum number of bytes used for in-memory caching of all reusable streams combined at any given time. When this limit is reached, file-based caching is used instead. Default is 1 GB.

maxStreamCacheSize​

The maximum number of bytes used for in-memory caching of a single reusable stream. When this per-stream limit is reached, file-based caching is used. Default is 100 MB.

metadataChecksummer​

Generates a checksum from a document's metadata to detect changes between crawl sessions. Set to null to disable metadata checksumming. Defaults to LastModifiedMetadataChecksummer for web crawls.

metadataDeduplicate​

When enabled, documents sharing the same metadata checksum within a crawl session are treated as duplicates and rejected. Requires a metadata checksummer to be configured. Not recommended unless checksums are guaranteed to be sufficiently unique.

metadataFetchSupport​

Controls how the HTTP HEAD request (metadata fetch) is handled. Supported values are DISABLED (no HEAD request, default), OPTIONAL (a failed HEAD does not stop processing if GET succeeds), and REQUIRED (a failed HEAD rejects the document).

Allowed Values​

  • DISABLED
  • OPTIONAL
  • REQUIRED

metadataFilters​

Filters applied to document metadata fields, allowing rejection of documents before content downloading (when using HTTP HEAD) or after downloading (when using HTTP GET only).

minProgressLoggingInterval​

The minimum amount of time to wait between each logging of crawling progress. A null value or a value below 1 second disables progress logging. Default is 30 seconds.

numThreads​

The maximum number of concurrent processing threads for the crawler. In clustered mode, this value applies per node. Default is 2.

orphansStrategy​

Determines what to do with previously crawled documents that are no longer found during the current crawl (orphans). Typical values are PROCESS, DELETE, or IGNORE.

Allowed Values​

  • PROCESS
  • DELETE
  • IGNORE

postImportConsumers​

A list of document consumers invoked after the document has been processed by the importer.

A field matcher identifying post-import metadata fields that contain URLs to be queued for crawling. Used for advanced cases where link extraction is deferred to the import phase.

postImportLinksKeep​

When enabled, the metadata fields matched by postImportLinks are retained on the document after their URLs have been queued. By default, those fields are deleted after processing.

preImportConsumers​

A list of document consumers invoked before the document is sent to the importer for content extraction and metadata enrichment.

recrawlableResolver​

Determines whether a previously crawled URL is ready to be re-crawled in the current session, based on elapsed time or other criteria. Defaults to GenericRecrawlableResolver.

referenceFilters​

Filters applied to document references (URLs or file paths) to control which ones are fetched. Documents whose references do not pass the filters are skipped.

robotsMetaProvider​

Provides robots metadata rules from HTTP response headers (e.g., X-Robots-Tag) for individual pages. Defaults to StandardRobotsMetaProvider. Set to null to disable robots metadata support.

robotsTxtProvider​

Provides robots.txt crawling rules for web sites. Defaults to StandardRobotsTxtProvider. Set to null to disable robots.txt support.

sitemapLocator​

Locates sitemaps for web sites by checking common paths or reading robots.txt. Defaults to GenericSitemapLocator. Set to null to disable automatic sitemap detection (only sitemap start references will be used).

sitemapResolver​

Resolves and processes sitemap files, extracting URLs for crawling. Defaults to GenericSitemapResolver. Set to null to disable all sitemap support.

spoiledReferenceStrategizer​

Determines how to handle references that repeatedly fail to be fetched or processed. Configures the strategy for dealing with such "spoiled" references.

startReferences​

The initial set of document references (URLs or file paths) from which the crawl begins.

startReferencesAsync​

When enabled, start references are provided and consumed asynchronously, allowing the crawl to begin before all start references are available. While this may speed up crawling, it may have an unexpected effect on the accuracy of the established page "depth". Usage of this option is only recommended when start URLs take a significant time to load.

startReferencesFiles​

One or more files containing lists of start references, one reference per line.

startReferencesProviders​

A list of pluggable providers that supply the initial references to crawl, used when start references come from a dynamic or external source.

startReferencesSitemaps​

One or more sitemap XML URLs to use as starting points for crawling. The sitemap resolver will process these to extract the URLs to crawl.

stopOnExceptions​

A list of fully qualified exception class names which, when encountered, will stop the crawler.

urlNormalizers​

One or more URL normalizers that convert URLs to a canonical form to prevent the same page from being crawled multiple times under different URL variations. Executed in order. Defaults to GenericUrlNormalizer.

urlScopeResolver​

Determines whether a discovered URL is within the crawl scope (e.g., confined to the same domain or site). Defaults to GenericUrlScopeResolver.

workDir​

The directory where working files (temporary data, queues, state) are stored during crawl execution.