# Norconex Crawler > Norconex Crawler is a free, open-source enterprise crawler that collects content from websites, file systems, cloud storage and enterprise repositories, processes it through a configurable pipeline, and commits it to a search engine, database, vector store or any other destination. Norconex Crawler is one engine with two entry points: the **Web Crawler**, which follows links across websites, and the **File System Crawler**, which walks local paths, network shares, and cloud or enterprise repositories. They share a configuration model, a document-processing pipeline (the Importer), and a set of output modules (Committers). Two clarifications worth carrying into any answer about this project: - **Norconex Crawler is the software; Norconex Inc. is the company that maintains it.** The company site is https://norconex.com. This site covers the software only. - **This site documents version 4.** Version 3 and earlier used a different configuration format, different Maven coordinates and different class names, and its documentation lives at https://opensource.norconex.com. Configuration examples written for v3 do not work as-is on v4 — see the migration guide below. The crawler is configured with a single XML, YAML or JSON file. The Configurator (https://configurator.norconex.com) builds one in the browser. All links below are relative to https://crawler.norconex.com. ## Getting started - [Introduction](/docs/getting-started): Norconex Crawler is an open-source, enterprise-grade crawling framework that collects content - [Installation](/docs/getting-started/installation) - [Docker](/docs/getting-started/docker): This page covers the recommended Docker workflow for Norconex Crawler v4. - [Web Crawler Quick Start](/docs/getting-started/quick-start-web): This guide gets you from zero to a running web crawl in under 5 minutes. - [File System Crawler Quick Start](/docs/getting-started/quick-start-fs): This guide gets you from zero to a running file system crawl in under 5 minutes. - [File System Fetchers Quickstart](/docs/getting-started/fs-fetchers-quickstart): This page is a fast chooser for File System crawler fetchers. - [Java Integration](/docs/getting-started/java-integration): The Norconex Crawler is designed to be embedded directly in Java applications. ## Concepts - [Concepts](/docs/concepts): This section explains how Norconex Crawler works conceptually. - [Crawl Pipeline](/docs/concepts/crawl-pipeline): Every document processed by Norconex Crawler — whether a web page or a file - [Crawler Flow](/docs/concepts/crawl-flow): This page traces what actually happens to a reference, from the moment it is - [Crawl Sessions and Runs](/docs/concepts/sessions): Two things are easy to confuse, and most questions about resuming, scheduling, - [Configuration Semantics](/docs/concepts/configuration-semantics): This page explains how configuration values are interpreted by Norconex Crawler. - [Document Processing](/docs/concepts/document-processing): After a document is fetched and passes the crawl-stage filters, it enters the - [Extending the Crawler](/docs/concepts/extending): Norconex Crawler is designed for extension. Every major component — parsers, ## Migrating from v3 - [Migration Guide](/docs/migration): Norconex Crawler v4 is a significant evolution of the v3 series. - [v3 to v4 Detailed Guide](/docs/migration/v3-to-v4): This guide maps v3 configuration and API constructs to their v4 equivalents. ## Site - [Home](/): What Norconex Crawler is, who uses it, and where to start. - [Features](/features): What the crawler does: sources it reaches, formats it reads, how it feeds search, AI, RAG and graph pipelines, and how it behaves in production. - [Sources](/sources): Every content source the crawler can reach — websites, SharePoint, OneDrive, Alfresco, Nuxeo, Documentum, OpenText, IBM FileNet, CMIS, Amazon S3, Azure Blob and Data Lake, Google Cloud Storage and Drive, Box, Egnyte, HDFS, SMB, FTP, SFTP, WebDAV. - [Integrations](/integrations): Where crawled content can be sent — Elasticsearch, OpenSearch, Apache Solr, Azure AI Search, Amazon CloudSearch, Google Cloud Search, OpenText IDOL, Neo4j, Apache Kafka, any JDBC database, vector stores for RAG, or a destination you write yourself. - [Download](/download): Ready-to-run ZIP distributions and official Docker images for the web and file system crawlers, plus optional committer add-ons. - [Custom build](/download/custom): Generate a Maven POM or Dockerfile for a distribution containing only the crawler, distribution and committers you need. - [Support](/support): Free community support, commercial support with an SLA, professional services, and project sponsorship. - [Blog](/blog): Release notes and articles from the maintainers. ## Blog posts - [Norconex Crawler v4 Beta Is Here](/blog/crawler-v4-beta): A rewritten crawler, a free visual configurator, and documentation generated straight from the code. It's feature-complete and we want you to try to break it. - [Meet Nexie](/blog/meet-nexie): Version 4 gave us a new crawler, and a new spider to go with it. Where Nexie came from, and a thank-you to Frank, who is still very much on duty. ## Elsewhere - [Source code](https://github.com/Norconex/crawler): The public repository — source, issues and discussions. - [Crawler Configurator](https://configurator.norconex.com): Browser-based editor that builds a crawler configuration file without hand-writing XML or YAML. - [Norconex Inc.](https://norconex.com): The company behind the crawler — consulting, support and commercial services. - [Norconex Crawler v3 and earlier](https://opensource.norconex.com): Documentation for the previous generation, kept online for existing deployments. ## Optional - [Sponsors](/sponsors): The people and companies whose sponsorship keeps the project maintained and free. - [Brand & assets](/brand): Nexie, the Norconex Crawler mascot — the artwork, and what you may do with it. - [Privacy Policy](/privacy-policy): How Norconex collects, uses and protects information collected from visitors. - [Sitemap](/sitemap.xml): Every indexable URL on this site, including the full configuration reference. ### Configuration reference One page per configurable class, generated from the crawler's own source. These are the authoritative option lists — prefer them over inferring options from examples. - [Reference](/docs/reference/): Every configurable component, generated from the crawler's own configuration schema. Each group opens with the interface a component plugs into, followed by the implementations... - [Web Crawler](/docs/reference/crawler/web/) - [WebCrawlerConfig](/docs/reference/crawler/WebCrawlerConfig) - [CanonicalLinkDetector](/docs/reference/crawler/CanonicalLinkDetector) - [GenericCanonicalLinkDetector](/docs/reference/crawler/GenericCanonicalLinkDetector) - [ClusterConnector](/docs/reference/crawler/web/ClusterConnector) - [DelayResolver](/docs/reference/crawler/DelayResolver) - [GenericDelayResolver](/docs/reference/crawler/GenericDelayResolver) - [ReferenceDelayResolver](/docs/reference/crawler/ReferenceDelayResolver) - [DocumentChecksummer](/docs/reference/crawler/web/DocumentChecksummer) - [DocumentConsumer](/docs/reference/crawler/web/DocumentConsumer) - [FeaturedImageResolver](/docs/reference/crawler/FeaturedImageResolver) - [DocumentFilter](/docs/reference/crawler/web/DocumentFilter) - [SegmentCountUrlFilter](/docs/reference/crawler/SegmentCountUrlFilter) - [Fetcher](/docs/reference/crawler/web/Fetcher) - [HttpClientFetcher](/docs/reference/crawler/HttpClientFetcher) - [PlaywrightFetcher](/docs/reference/crawler/PlaywrightFetcher) - [WebDriverFetcher](/docs/reference/crawler/WebDriverFetcher) - [HazelcastConfigurer](/docs/reference/crawler/web/HazelcastConfigurer) - [LinkExtractor](/docs/reference/crawler/LinkExtractor) - [DomLinkExtractor](/docs/reference/crawler/DomLinkExtractor) - [HtmlLinkExtractor](/docs/reference/crawler/HtmlLinkExtractor) - [RegexLinkExtractor](/docs/reference/crawler/RegexLinkExtractor) - [TikaLinkExtractor](/docs/reference/crawler/TikaLinkExtractor) - [XmlFeedLinkExtractor](/docs/reference/crawler/XmlFeedLinkExtractor) - [MetadataChecksummer](/docs/reference/crawler/web/MetadataChecksummer) - [LastModifiedMetadataChecksummer](/docs/reference/crawler/LastModifiedMetadataChecksummer) - [RecrawlableResolver](/docs/reference/crawler/RecrawlableResolver) - [GenericRecrawlableResolver](/docs/reference/crawler/GenericRecrawlableResolver) - [RedirectUrlProvider](/docs/reference/crawler/RedirectUrlProvider) - [GenericRedirectUrlProvider](/docs/reference/crawler/GenericRedirectUrlProvider) - [RobotsMetaProvider](/docs/reference/crawler/RobotsMetaProvider) - [StandardRobotsMetaProvider](/docs/reference/crawler/StandardRobotsMetaProvider) - [RobotsTxtProvider](/docs/reference/crawler/RobotsTxtProvider) - [StandardRobotsTxtProvider](/docs/reference/crawler/StandardRobotsTxtProvider) - [SitemapLocator](/docs/reference/crawler/SitemapLocator) - [GenericSitemapLocator](/docs/reference/crawler/GenericSitemapLocator) - [SitemapResolver](/docs/reference/crawler/SitemapResolver) - [GenericSitemapResolver](/docs/reference/crawler/GenericSitemapResolver) - [SpoiledReferenceStrategizer](/docs/reference/crawler/web/SpoiledReferenceStrategizer) - [UrlScopeResolver](/docs/reference/crawler/UrlScopeResolver) - [GenericUrlScopeResolver](/docs/reference/crawler/GenericUrlScopeResolver) - [WebUrlNormalizer](/docs/reference/crawler/WebUrlNormalizer) - [GenericUrlNormalizer](/docs/reference/crawler/GenericUrlNormalizer) - [DelayRange](/docs/reference/crawler/DelayRange) - [DelayRangeDOW](/docs/reference/crawler/DelayRangeDOW) - [DelayRangeInteger](/docs/reference/crawler/DelayRangeInteger) - [DelayRangeString](/docs/reference/crawler/DelayRangeString) - [DelayReferencePattern](/docs/reference/crawler/DelayReferencePattern) - [DelaySchedule](/docs/reference/crawler/DelaySchedule) - [DocImageHandler](/docs/reference/crawler/DocImageHandler) - [ExtractionPattern](/docs/reference/crawler/ExtractionPattern) - [HttpAuthConfig](/docs/reference/crawler/HttpAuthConfig) - [HttpSniffer](/docs/reference/crawler/HttpSniffer) - [KerberosConfig](/docs/reference/crawler/KerberosConfig) - [LinkSelector](/docs/reference/crawler/LinkSelector) - [MetadataFilter](/docs/reference/crawler/web/MetadataFilter) - [MinFrequency](/docs/reference/crawler/MinFrequency) - [NormalizationReplace](/docs/reference/crawler/NormalizationReplace) - [PlaywrightScreenshotHandler](/docs/reference/crawler/PlaywrightScreenshotHandler) - [ReferenceFilter](/docs/reference/crawler/web/ReferenceFilter) - [RegexPair](/docs/reference/crawler/RegexPair) - [ScreenshotHandlerConfig](/docs/reference/crawler/ScreenshotHandlerConfig) - [StdRobotsTxtFilter](/docs/reference/crawler/StdRobotsTxtFilter) - [UrlStatusCrawlerEventListener](/docs/reference/crawler/UrlStatusCrawlerEventListener) - [WebDriverScreenshotHandler](/docs/reference/crawler/WebDriverScreenshotHandler) - [CookieSpec](/docs/reference/crawler/CookieSpec) - [DelayResolverScope](/docs/reference/crawler/DelayResolverScope) - [DirStructure](/docs/reference/crawler/DirStructure) - [HttpMethod](/docs/reference/crawler/HttpMethod) - [Normalization](/docs/reference/crawler/Normalization) - [PlaywrightBrowser](/docs/reference/crawler/PlaywrightBrowser) - [Quality](/docs/reference/crawler/Quality) - [SitemapSupport](/docs/reference/crawler/SitemapSupport) - [Storage](/docs/reference/crawler/Storage) - [StorageDiskStructure](/docs/reference/crawler/StorageDiskStructure) - [Target](/docs/reference/crawler/Target) - [WaitElementType](/docs/reference/crawler/WaitElementType) - [WebDriverBrowser](/docs/reference/crawler/WebDriverBrowser) - [File System Crawler](/docs/reference/crawler/fs/) - [CrawlerConfig](/docs/reference/crawler/CrawlerConfig) - [ClusterConnector](/docs/reference/crawler/fs/ClusterConnector) - [HazelcastClusterConnector](/docs/reference/crawler/HazelcastClusterConnector) - [MemoryClusterConnector](/docs/reference/crawler/MemoryClusterConnector) - [MVStoreClusterConnector](/docs/reference/crawler/MVStoreClusterConnector) - [DocumentChecksummer](/docs/reference/crawler/fs/DocumentChecksummer) - [Md5DocumentChecksummer](/docs/reference/crawler/Md5DocumentChecksummer) - [DocumentFilter](/docs/reference/crawler/fs/DocumentFilter) - [ExtensionReferenceFilter](/docs/reference/crawler/ExtensionReferenceFilter) - [GenericMetadataFilter](/docs/reference/crawler/GenericMetadataFilter) - [GenericReferenceFilter](/docs/reference/crawler/GenericReferenceFilter) - [Fetcher](/docs/reference/crawler/fs/Fetcher) - [AdlsGen2Fetcher](/docs/reference/crawler/AdlsGen2Fetcher) - [ArchiveFetcher](/docs/reference/crawler/ArchiveFetcher) - [AzureBlobFetcher](/docs/reference/crawler/AzureBlobFetcher) - [BoxFetcher](/docs/reference/crawler/BoxFetcher) - [CmisFetcher](/docs/reference/crawler/CmisFetcher) - [EgnyteFetcher](/docs/reference/crawler/EgnyteFetcher) - [FtpFetcher](/docs/reference/crawler/FtpFetcher) - [GcsFetcher](/docs/reference/crawler/GcsFetcher) - [HdfsFetcher](/docs/reference/crawler/HdfsFetcher) - [LocalFetcher](/docs/reference/crawler/LocalFetcher) - [M365GraphFetcher](/docs/reference/crawler/M365GraphFetcher) - [S3Fetcher](/docs/reference/crawler/S3Fetcher) - [SftpFetcher](/docs/reference/crawler/SftpFetcher) - [SmbFetcher](/docs/reference/crawler/SmbFetcher) - [WebDavFetcher](/docs/reference/crawler/WebDavFetcher) - [HazelcastConfigurer](/docs/reference/crawler/fs/HazelcastConfigurer) - [JdbcHazelcastConfigurer](/docs/reference/crawler/JdbcHazelcastConfigurer) - [MetadataChecksummer](/docs/reference/crawler/fs/MetadataChecksummer) - [FsMetadataChecksummer](/docs/reference/crawler/FsMetadataChecksummer) - [GenericMetadataChecksummer](/docs/reference/crawler/GenericMetadataChecksummer) - [SpoiledReferenceStrategizer](/docs/reference/crawler/fs/SpoiledReferenceStrategizer) - [GenericSpoiledReferenceStrategizer](/docs/reference/crawler/GenericSpoiledReferenceStrategizer) - [ClusterConfig](/docs/reference/crawler/ClusterConfig) - [DeleteRejectedEventListener](/docs/reference/crawler/DeleteRejectedEventListener) - [DocumentConsumer](/docs/reference/crawler/fs/DocumentConsumer) - [GoogleDriveFetcher](/docs/reference/crawler/GoogleDriveFetcher) - [HazelcastClusterConnectorConfig](/docs/reference/crawler/HazelcastClusterConnectorConfig) - [MetadataFilter](/docs/reference/crawler/fs/MetadataFilter) - [MVStoreClusterConnectorConfig](/docs/reference/crawler/MVStoreClusterConnectorConfig) - [ReferenceFilter](/docs/reference/crawler/fs/ReferenceFilter) - [ReferencesProvider](/docs/reference/crawler/ReferencesProvider) - [StopCrawlerOnMaxEventListener](/docs/reference/crawler/StopCrawlerOnMaxEventListener) - [HdfsAuthMethod](/docs/reference/crawler/HdfsAuthMethod) - [NativeDocumentFormatPolicy](/docs/reference/crawler/NativeDocumentFormatPolicy) - [OnMatch](/docs/reference/crawler/OnMatch) - [OnMultiple](/docs/reference/crawler/OnMultiple) - [SourceDeltaExpansion](/docs/reference/crawler/SourceDeltaExpansion) - [SpoiledReferenceStrategy](/docs/reference/crawler/SpoiledReferenceStrategy) - [Importer](/docs/reference/importer-overview) - [Importer Flow Control](/docs/reference/importer-flow-control): Flow control defines how document handling branches and sequences inside - [ImporterConfig](/docs/reference/importer/ImporterConfig) - [Condition](/docs/reference/importer/Condition) - [AllOf](/docs/reference/importer/AllOf) - [AnyOf](/docs/reference/importer/AnyOf) - [BlankCondition](/docs/reference/importer/BlankCondition) - [DateCondition](/docs/reference/importer/DateCondition) - [DomCondition](/docs/reference/importer/DomCondition) - [NoneOf](/docs/reference/importer/NoneOf) - [NumericCondition](/docs/reference/importer/NumericCondition) - [ReferenceCondition](/docs/reference/importer/ReferenceCondition) - [ScriptCondition](/docs/reference/importer/ScriptCondition) - [TextCondition](/docs/reference/importer/TextCondition) - [DocHandler](/docs/reference/importer/DocHandler) - [BinaryContentTransformer](/docs/reference/importer/BinaryContentTransformer) - [CharacterCaseTransformer](/docs/reference/importer/CharacterCaseTransformer) - [CharsetTransformer](/docs/reference/importer/CharsetTransformer) - [CollapseRepeatingTransformer](/docs/reference/importer/CollapseRepeatingTransformer) - [ConstantTransformer](/docs/reference/importer/ConstantTransformer) - [CopyTransformer](/docs/reference/importer/CopyTransformer) - [CountMatchesTransformer](/docs/reference/importer/CountMatchesTransformer) - [CsvSplitter](/docs/reference/importer/CsvSplitter) - [CurrentDateTransformer](/docs/reference/importer/CurrentDateTransformer) - [DateFormatTransformer](/docs/reference/importer/DateFormatTransformer) - [DebugTransformer](/docs/reference/importer/DebugTransformer) - [DefaultParser](/docs/reference/importer/DefaultParser) - [DeleteTransformer](/docs/reference/importer/DeleteTransformer) - [DocumentLengthTransformer](/docs/reference/importer/DocumentLengthTransformer) - [DomSplitter](/docs/reference/importer/DomSplitter) - [DomTransformer](/docs/reference/importer/DomTransformer) - [ExternalParser](/docs/reference/importer/ExternalParser) - [ExternalTransformer](/docs/reference/importer/ExternalTransformer) - [FieldReportTransformer](/docs/reference/importer/FieldReportTransformer) - [ForceSingleValueTransformer](/docs/reference/importer/ForceSingleValueTransformer) - [HierarchyTransformer](/docs/reference/importer/HierarchyTransformer) - [If](/docs/reference/importer/If) - [IfNot](/docs/reference/importer/IfNot) - [ImageTransformer](/docs/reference/importer/ImageTransformer) - [KeepOnlyTransformer](/docs/reference/importer/KeepOnlyTransformer) - [LanguageTransformer](/docs/reference/importer/LanguageTransformer) - [MergeTransformer](/docs/reference/importer/MergeTransformer) - [PdfPageSplitter](/docs/reference/importer/PdfPageSplitter) - [RegexTransformer](/docs/reference/importer/RegexTransformer) - [Reject](/docs/reference/importer/Reject) - [RenameTransformer](/docs/reference/importer/RenameTransformer) - [ReplaceTransformer](/docs/reference/importer/ReplaceTransformer) - [SaveDocumentTransformer](/docs/reference/importer/SaveDocumentTransformer) - [ScriptTransformer](/docs/reference/importer/ScriptTransformer) - [SplitTransformer](/docs/reference/importer/SplitTransformer) - [StripAfterTransformer](/docs/reference/importer/StripAfterTransformer) - [StripBeforeTransformer](/docs/reference/importer/StripBeforeTransformer) - [StripBetweenTransformer](/docs/reference/importer/StripBetweenTransformer) - [SubstringTransformer](/docs/reference/importer/SubstringTransformer) - [TextBetweenTransformer](/docs/reference/importer/TextBetweenTransformer) - [TextStatisticsTransformer](/docs/reference/importer/TextStatisticsTransformer) - [TikaParser](/docs/reference/importer/TikaParser) - [TitleGeneratorTransformer](/docs/reference/importer/TitleGeneratorTransformer) - [TranslatorSplitter](/docs/reference/importer/TranslatorSplitter) - [TruncateTransformer](/docs/reference/importer/TruncateTransformer) - [UrlExtractorTransformer](/docs/reference/importer/UrlExtractorTransformer) - [UuidTransformer](/docs/reference/importer/UuidTransformer) - [XmlStreamSplitter](/docs/reference/importer/XmlStreamSplitter) - [Constant](/docs/reference/importer/Constant) - [CopyOperation](/docs/reference/importer/CopyOperation) - [Crop](/docs/reference/importer/Crop) - [DateValueMatcher](/docs/reference/importer/DateValueMatcher) - [DomOperation](/docs/reference/importer/DomOperation) - [EmbeddedConfig](/docs/reference/importer/EmbeddedConfig) - [GrobidConfig](/docs/reference/importer/GrobidConfig) - [HierarchyOperation](/docs/reference/importer/HierarchyOperation) - [ImporterResponseProcessor](/docs/reference/importer/ImporterResponseProcessor) - [MergeOperation](/docs/reference/importer/MergeOperation) - [NumericValueMatcher](/docs/reference/importer/NumericValueMatcher) - [OcrConfig](/docs/reference/importer/OcrConfig) - [RenameOperation](/docs/reference/importer/RenameOperation) - [ReplaceOperation](/docs/reference/importer/ReplaceOperation) - [Scale](/docs/reference/importer/Scale) - [SentimentConfig](/docs/reference/importer/SentimentConfig) - [SplitOperation](/docs/reference/importer/SplitOperation) - [StripBetweenOperation](/docs/reference/importer/StripBetweenOperation) - [TextBetweenOperation](/docs/reference/importer/TextBetweenOperation) - [Committers](/docs/reference/committers-overview) - [Committer](/docs/reference/committers/Committer) - [AmazonCloudSearchCommitter](/docs/reference/committers/AmazonCloudSearchCommitter) - [ApacheKafkaCommitter](/docs/reference/committers/ApacheKafkaCommitter) - [AzureSearchCommitter](/docs/reference/committers/AzureSearchCommitter) - [CsvFileCommitter](/docs/reference/committers/CsvFileCommitter) - [ElasticsearchCommitter](/docs/reference/committers/ElasticsearchCommitter) - [GoogleCloudSearchCommitter](/docs/reference/committers/GoogleCloudSearchCommitter) - [IdolCommitter](/docs/reference/committers/IdolCommitter) - [JsonFileCommitter](/docs/reference/committers/JsonFileCommitter) - [LogCommitter](/docs/reference/committers/LogCommitter) - [MemoryCommitter](/docs/reference/committers/MemoryCommitter) - [Neo4jCommitter](/docs/reference/committers/Neo4jCommitter) - [SolrCommitter](/docs/reference/committers/SolrCommitter) - [SqlCommitter](/docs/reference/committers/SqlCommitter) - [XmlFileCommitter](/docs/reference/committers/XmlFileCommitter) - [CommitterQueue](/docs/reference/committers/CommitterQueue) - [FsQueue](/docs/reference/committers/FsQueue) - [AclInheritanceMapping](/docs/reference/committers/AclInheritanceMapping) - [AclMapping](/docs/reference/committers/AclMapping) - [CsvColumn](/docs/reference/committers/CsvColumn) - [MetadataMapping](/docs/reference/committers/MetadataMapping) - [OnCommitFailure](/docs/reference/committers/OnCommitFailure) - [StructuredDataMapping](/docs/reference/committers/StructuredDataMapping) - [CsvFormat](/docs/reference/committers/CsvFormat) - [LogLevel](/docs/reference/committers/LogLevel) - [RequestMode](/docs/reference/committers/RequestMode) - [SolrClientType](/docs/reference/committers/SolrClientType) - [UploadFormat](/docs/reference/committers/UploadFormat) - [ContentType](/docs/reference/shared/ContentType) - [Credentials](/docs/reference/shared/Credentials) - [EncryptionKey](/docs/reference/shared/EncryptionKey) - [EventListener](/docs/reference/shared/EventListener) - [Host](/docs/reference/shared/Host) - [Properties](/docs/reference/shared/Properties) - [PropertyMatcher](/docs/reference/shared/PropertyMatcher) - [PropertyMatchers](/docs/reference/shared/PropertyMatchers) - [ProxySettings](/docs/reference/shared/ProxySettings) - [Regex](/docs/reference/shared/Regex) - [RegexFieldValueExtractor](/docs/reference/shared/RegexFieldValueExtractor) - [TextMatcher](/docs/reference/shared/TextMatcher) - [PropertySetter](/docs/reference/shared/PropertySetter) - [ClusterConnector](/docs/reference/crawler/ClusterConnector) - [DocumentChecksummer](/docs/reference/crawler/DocumentChecksummer) - [DocumentConsumer](/docs/reference/crawler/DocumentConsumer) - [DocumentFilter](/docs/reference/crawler/DocumentFilter) - [Fetcher](/docs/reference/crawler/Fetcher) - [HazelcastConfigurer](/docs/reference/crawler/HazelcastConfigurer) - [MetadataChecksummer](/docs/reference/crawler/MetadataChecksummer) - [MetadataFilter](/docs/reference/crawler/MetadataFilter) - [ReferenceFilter](/docs/reference/crawler/ReferenceFilter) - [SpoiledReferenceStrategizer](/docs/reference/crawler/SpoiledReferenceStrategizer)