Reference

Every configurable component, generated from the crawler's own configuration schema. Each group opens with the interface a component plugs into, followed by the implementations you can name in its class.

Crawler

• WebCrawlerConfig: The top-level configuration for a web crawler.
• CrawlerConfig: The top-level configuration for a crawler.

CanonicalLinkDetector

Identifies the possible canonical URL of a document. Non-canonical documents are rejected in favor of canonical ones. See CanonicalLinkDetector for the shared configuration.

• GenericCanonicalLinkDetector: Replace all URLs encountered with their canonical version (when applicable).

ClusterConnector

Supported implementations for Cluster Connector. See ClusterConnector for the shared configuration.

• HazelcastClusterConnector: Connects distributed crawler nodes using Hazelcast.
• MemoryClusterConnector: Crawl state held in memory only, with nothing written to disk.
• MVStoreClusterConnector: File-backed crawl state using H2's MVStore engine. The default connector.

DelayResolver

Supported implementations for Delay Resolver. See DelayResolver for the shared configuration.

• GenericDelayResolver: Apply delay per time schedule.
• ReferenceDelayResolver: Apply different delays per different URL patterns.

DocumentChecksummer

Supported implementations for Document Checksummer. See DocumentChecksummer for the shared configuration.

• Md5DocumentChecksummer: Generate a MD5 checksum from content (default) or fields.

DocumentConsumer

Supported implementations for Document Consumer. See DocumentConsumer for the shared configuration.

• FeaturedImageResolver: Extracts and optionally scales a featured image from crawled web pages.

DocumentFilter

Supported implementations for Document Filter. See DocumentFilter for the shared configuration.

• ExtensionReferenceFilter: Filter by file extension (if present on last segment).
• GenericMetadataFilter: Filter by one or more matching metadata field values.
• GenericReferenceFilter: Filter references by matching text expression.
• SegmentCountUrlFilter: Filter by the number of reference segments.

Fetcher

Supported implementations for Fetcher. See Fetcher for the shared configuration.

• AdlsGen2Fetcher: Azure Data Lake Storage Gen2 fetcher (abfs:// and abfss://).
• ArchiveFetcher: Fetches documents contained within archive files (ZIP, TAR, etc.).
• AzureBlobFetcher: Azure Blob Storage fetcher (azblob:// and azureblob://).
• BoxFetcher: Box fetcher (box://) for filesystem-like traversal and content download through Box APIs.
• CmisFetcher: CMIS-enabled Content Management Systems (CMS) fetcher (Atom).
• EgnyteFetcher: Egnyte fetcher (egnyte://) for filesystem-like traversal and content download through Egnyte APIs.
• FtpFetcher: File Transfer Protocol (FTP and FTPS) fetcher.
• GcsFetcher: Google Cloud Storage fetcher (gs://).
• HdfsFetcher: Apache Hadoop File System (HDFS) fetcher.
• HttpClientFetcher: Fetches web documents using an HTTP client with authentication and SSL support.
• LocalFetcher: Local file system fetcher.
• M365GraphFetcher: Microsoft 365 Graph fetcher for SharePoint Online and OneDrive for Business.
• PlaywrightFetcher: Uses Microsoft Playwright for browser-based crawling, driving real browser engines (Chromium, Firefox, WebKit) across Windows, macOS, and Linux. Unlike WebDriverFetcher, no exte...
• S3Fetcher: Amazon S3 fetcher (s3://).
• SftpFetcher: Secure File Transfer Protocol (SFTP) fetcher.
• SmbFetcher: Server Message Block (SMB, CIFS, Samba, Windows share) fetcher.
• WebDavFetcher: Web Distributed Authoring and Versioning (WebDAV) fetcher.
• WebDriverFetcher: Fetches dynamically generated web pages using a WebDriver-based browser.

HazelcastConfigurer

Supported implementations for Hazelcast Configurer. See HazelcastConfigurer for the shared configuration.

• JdbcHazelcastConfigurer: Configures Hazelcast for distributed crawling using a JDBC-backed data store.

LinkExtractor

Supported implementations for Link Extractor. See LinkExtractor for the shared configuration.

• DomLinkExtractor: Extracts links from HTML/XML documents using CSS/jQuery-style selectors.
• HtmlLinkExtractor: Extracts hyperlinks from HTML documents by scanning tag attributes.
• RegexLinkExtractor: Extracts links from documents using regular expression patterns.
• TikaLinkExtractor: Extracts links from documents using Apache Tika.
• XmlFeedLinkExtractor: Extracts links from RSS and Atom XML feed documents.

MetadataChecksummer

Supported implementations for Metadata Checksummer. See MetadataChecksummer for the shared configuration.

• FsMetadataChecksummer: Generates a metadata checksum for file system documents based on file attributes.
• GenericMetadataChecksummer: Use one or more matching metadata field values.
• LastModifiedMetadataChecksummer: Uses the exact value of the "Last-Modified" HTTP header field.

RecrawlableResolver

Supported implementations for Recrawlable Resolver. See RecrawlableResolver for the shared configuration.

• GenericRecrawlableResolver: Determines whether a previously crawled document should be re-crawled.

RedirectUrlProvider

Resolves the target URL from HTTP redirect responses. See RedirectUrlProvider for the shared configuration.

• GenericRedirectUrlProvider: Resolves the target URL from HTTP redirect responses.

RobotsMetaProvider

Supported implementations for Robots Meta Provider. See RobotsMetaProvider for the shared configuration.

• StandardRobotsMetaProvider: Reads robots directives from HTML meta tags and HTTP response headers.

RobotsTxtProvider

Supported implementations for Robots Txt Provider. See RobotsTxtProvider for the shared configuration.

• StandardRobotsTxtProvider: Fetches and enforces robots.txt crawl rules from web sites.

SitemapLocator

Supported implementations for Sitemap Locator. See SitemapLocator for the shared configuration.

• GenericSitemapLocator: Locates sitemap files for crawled websites.

SitemapResolver

Supported implementations for Sitemap Resolver. See SitemapResolver for the shared configuration.

• GenericSitemapResolver: Parses sitemap XML files to extract URLs for crawling.

SpoiledReferenceStrategizer

Supported implementations for Spoiled Reference Strategizer. See SpoiledReferenceStrategizer for the shared configuration.

• GenericSpoiledReferenceStrategizer: Generic strategy mapping between each references that have turned "bad" and their state.

UrlScopeResolver

Supported implementations for Url Scope Resolver. See UrlScopeResolver for the shared configuration.

• GenericUrlScopeResolver: Generic implementation that should satisfy most common site adherence scenarios.

WebUrlNormalizer

Supported implementations for Web Url Normalizer. See WebUrlNormalizer for the shared configuration.

• GenericUrlNormalizer: Generic implementation that should satisfy most URL normalization needs.

Other types

• ClusterConfig: Cluster configuration for distributed crawling.
• DelayRange: A start/end range used for time-based delay scheduling.
• DelayRangeDOW: A day-of-week range with start and end values for delay scheduling.
• DelayRangeInteger: An integer-based range with start and end values for delay scheduling.
• DelayRangeString: A string-based time range with start and end values for delay scheduling.
• DelayReferencePattern: URL pattern
• DelaySchedule: Dictates what delay to use at the specified time ranges. Range start value can be greater than the end value.
• DeleteRejectedEventListener: Sends deletion requests to the committer when documents are rejected or not found.
• DocImageHandler: Defines how document images are saved and referenced during crawling.
• ExtractionPattern: A regex pattern for extracting metadata from external application output.
• FsCrawlerConfig: Configuration schema for Fs Crawler Config.
• GoogleDriveFetcher: Google Drive fetcher (gdrive://) for Google Workspace My Drive and Shared Drives.
• HazelcastClusterConnectorConfig: Configuration settings for the Hazelcast-based cluster connector.
• HttpAuthConfig: HTTP authentication configuration for accessing protected websites.
• HttpSniffer: A local proxy that intercepts browser HTTP traffic for inspection.
• KerberosConfig: Kerberos/SPNEGO authentication settings for HDFS WebHDFS access.
• LinkSelector: A CSS/jQuery-style selector defining which element attribute to extract as a link URL.
• MetadataFilter: Supported implementations for Metadata Filter.
• MinFrequency: A minimum recrawl frequency rule for documents matching a given pattern.
• MVStoreClusterConnectorConfig: Storage tuning for the MVStore cluster connector.
• NormalizationReplace: Replace URL text.
• PlaywrightScreenshotHandler: Captures page screenshots through Playwright, as a whole viewport or a single element.
• ReferenceFilter: Supported implementations for Reference Filter.
• ReferencesProvider: Provides start references to the crawler from an external source.
• RegexPair: A pair of start and end regular expressions defining a text region boundary.
• ScreenshotHandlerConfig: Where page screenshots are stored, in what format, and what part of the page is captured.
• StdRobotsTxtFilter: A reference filter that enforces robots.txt rules on crawled URLs.
• StopCrawlerOnMaxEventListener: The following example will stop the crawler when the sum of committed documents (upserts + deletions) reaches 100.
• UrlStatusCrawlerEventListener: Records URLs and their HTTP status codes to a CSV report during a crawl.
• WebDriverScreenshotHandler: Captures page screenshots through Selenium WebDriver, as a whole page or a single element.

Enumerations

• CookieSpec: Supported cookie specification policies for HTTP cookie handling.
• DelayResolverScope: Controls the scope at which crawl delays are applied.
• DirStructure: Defines the directory structure convention used for on-disk storage.
• HdfsAuthMethod: Allowed values for Hdfs Auth Method.
• HttpMethod: Supported HTTP methods for fetching documents.
• NativeDocumentFormatPolicy: Allowed values for Native Document Format Policy.
• Normalization: Supported URL normalization operations.
• OnMatch: Action to take when a filter or condition matches.
• OnMultiple: Action to take when multiple values are found for a field operation.
• PlaywrightBrowser: Supported browser engines for Playwright-based fetching.
• Quality: Supported image quality levels for document image processing.
• SitemapSupport: Controls how sitemap files are used during a crawl.
• SourceDeltaExpansion: Allowed values for Source Delta Expansion.
• SpoiledReferenceStrategy: Supported strategies for handling references that can no longer be fetched.
• Storage: Supported storage backend types for the crawler data store.
• StorageDiskStructure: Supported on-disk directory structure layouts for the storage engine.
• Target: Specifies the target destination for a field operation or extraction.
• WaitElementType: Supported element wait condition types for WebDriver-based fetching.
• WebDriverBrowser: Supported browser types for WebDriver-based fetching.

Importer

• ImporterConfig: Configures the document importing pipeline with handlers for parsing, transforming, and filtering documents.

Condition

Supported implementations for Condition. See Condition for the shared configuration.

• AllOf: A compound condition that is true only when all child conditions are true.
• AnyOf: A compound condition that is true when any one child condition is true.
• BlankCondition: Evaluates to true when matching document fields are blank or empty.
• DateCondition: Evaluates to true when a document date field falls within a specified date range.
• DomCondition: Evaluates to true when a CSS selector matches an element in an HTML or XML document.
• NoneOf: A compound condition that is true only when none of its child conditions are true.
• NumericCondition: Evaluates to true when a document field value satisfies a numeric comparison.
• ReferenceCondition: Evaluates to true when a document's reference matches a configured text expression.
• ScriptCondition: Evaluates a custom script that returns true or false for flow control decisions.
• TextCondition: Evaluates to true when document text content or field values match a configured expression.

DocHandler

Supported implementations for Doc Handler. See DocHandler for the shared configuration.

• BinaryContentTransformer: Captures the document's original, unparsed content and stores it, Base64-encoded, in a metadata field.
• CharacterCaseTransformer: Changes the character case.
• CharsetTransformer: Converts a document's character encoding.
• CollapseRepeatingTransformer: Reduces specified consecutive characters or strings
• ConstantTransformer: Define and add constant values to documents.
• CopyTransformer: Copies metadata fields to new fields
• CountMatchesTransformer: Counts the occurrences of matches within a field.
• CsvSplitter: Split files with Coma-Separated values
• CurrentDateTransformer: Adds the current computer UTC date to the specified field
• DateFormatTransformer: Converts a date from its current format to a specified format.
• DebugTransformer: Logs document details for debugging during processing.
• DefaultParser: Generic document parser for raw content passthrough or text decoding
• DeleteTransformer: Delete the metadata fields provided
• DocumentLengthTransformer: Adds the document length to a specified field
• DomSplitter: Splits HTML, XHTML, or XML document on elements matching a given selector.
• DomTransformer: Extract the value of one or more elements or attributes into a target field, or delete matching elements
• ExternalParser: Parses and extracts text from a file using an external application
• ExternalTransformer: Transforms a document using an external application to do so
• FieldReportTransformer: Generates CSV reports of discovered fields.
• ForceSingleValueTransformer: Forces a metadata field to be single-value
• HierarchyTransformer: Adds hierarchical tags to documents using a delimiter for structured categorization.
• If: Executes handlers conditionally when a condition evaluates to true.
• IfNot: Executes handlers conditionally when a condition evaluates to false.
• ImageTransformer: Transforms an image using common image operations.
• KeepOnlyTransformer: Retains specified metadata fields and removes all others.
• LanguageTransformer: Tags documents with detected language metadata
• MergeTransformer: Merge multiple metadata fields into a single one.
• PdfPageSplitter: Split PDFs pages so each pages are treated as individual documents.
• RegexTransformer: Extracts field names and their values with regular expression
• Reject: Rejects a document from further processing
• RenameTransformer: Rename metadata fields to different names.
• ReplaceTransformer: Replaces an existing metadata value with another one.
• SaveDocumentTransformer: Saves document content or metadata to a specified directory.
• ScriptTransformer: Transformer incoming documents using a scripting language.
• SplitTransformer: Splits an existing metadata value into multiple values
• StripAfterTransformer: Removes all content following the first occurrence of the specified pattern.
• StripBeforeTransformer: Removes all content preceding the first occurrence of the specified pattern.
• StripBetweenTransformer: Removes any content found between a matching start and end strings.
• SubstringTransformer: Keep a substring of the content matching a begin and end character indexes.
• TextBetweenTransformer: Extracts and add values found between a matching start and end strings to a document metadata field
• TextStatisticsTransformer: Document content analysis and statistical metadata generation.
• TikaParser: Provides direct access to Apache Tika's parsing capabilities
• TitleGeneratorTransformer: Title Generation from Document Content
• TranslatorSplitter: Translate documents using one of the supported translation API.
• TruncateTransformer: Truncates field values
• UrlExtractorTransformer: Extracts unique URLs matching patterns
• UuidTransformer: Generates a random Universally unique identifier
• XmlStreamSplitter: Splits XML document on a specific element.

Other types

• Constant: A name-value constant entry added to documents by ConstantTransformer.
• CopyOperation: A single field copy operation specifying source fields and a target field.
• Crop: Cropping dimensions (x, y, width, height) for an image transformation.
• DateValueMatcher: A date value with a comparison operator used as a date range boundary.
• DomOperation: A single DOM extraction or deletion operation for the DomTransformer.
• EmbeddedConfig: Controls how embedded documents within containers are handled during parsing.
• GrobidConfig: Configuration for GROBID structured extraction from scientific documents.
• HierarchyOperation: A single hierarchy expansion operation for the HierarchyTransformer.
• ImporterResponseProcessor: A custom processor that modifies or acts on the importer response after all handlers have run.
• MergeOperation: A single merge operation combining matching source fields into a target field.
• NumericValueMatcher: A numeric value with a comparison operator used as a numeric range boundary.
• OcrConfig: Configuration for Tesseract OCR text extraction from images and image-containing documents.
• RenameOperation: A single field rename operation mapping source field names to a new name.
• ReplaceOperation: A single value replacement operation for the ReplaceTransformer.
• Scale: Image scaling parameters (width, height, factor, stretch) for an image transformation.
• SentimentConfig: Configuration for Tika sentiment analysis of document content.
• SplitOperation: A single value split operation for the SplitTransformer.
• StripBetweenOperation: A start/end delimiter pair defining a text region to strip.
• TextBetweenOperation: A start/end delimiter pair defining a text region to extract and store.

Committers

Committer

Supported implementations for Committer. See Committer for the shared configuration.

• AmazonCloudSearchCommitter: Commit documents to Amazon CloudSearch.
• ApacheKafkaCommitter: Commit documents to Apache Kafka.
• AzureSearchCommitter: Commit documents to Microsoft Azure Search.
• CsvFileCommitter: Local comma-separated values files.
• ElasticsearchCommitter: Commit documents to Elasticsearch or Amazon OpenSearch.
• GoogleCloudSearchCommitter: Commit documents to Google Cloud Search.
• IdolCommitter: Commit documents to OpenText IDOL search engine.
• JsonFileCommitter: Local files of JSON format.
• LogCommitter: Logs data associated with every document crawled without saving them.
• MemoryCommitter: For programmers. Not for production use.
• Neo4jCommitter: Commit documents to a Neo4j graph database.
• SolrCommitter: Commit documents to Apache Solr search engine.
• SqlCommitter: Commit documents to an SQL table.
• XmlFileCommitter: Local files of XML format.

CommitterQueue

Supported implementations for Committer Queue. See CommitterQueue for the shared configuration.

• FsQueue: A local file system queue for storing committer requests as zip files.

Other types

• AclInheritanceMapping: Declares which document field holds the parent item reference used for Google Cloud Search ACL inheritance.
• AclMapping: Maps a document field to a Google Cloud Search ACL principal.
• CsvColumn: A CSV file column.
• MetadataMapping: Maps a document field to one of Google Cloud Search's standard item metadata fields.
• OnCommitFailure: Controls retry and error handling behavior when a committer fails.
• StructuredDataMapping: Declares the Google Cloud Search value type to use for a document field sent as structured data.

Enumerations

• CsvFormat: Predefined CSV format presets for common applications and standards.
• LogLevel: Supported logging severity levels.
• RequestMode: Allowed values for Request Mode.
• SolrClientType: Supported Solr client connection types.
• UploadFormat: Allowed values for Upload Format.

Shared Types

Other types

• ContentType: A MIME content type used for filtering and restricting handler applicability.
• Credentials: User credentials with an optionally encrypted password. To be encrypted, there needs to be an encryption key. Without one, the password is assumed not to be encrypted.
• EncryptionKey: Passwords can be encrypted using the EncryptionUtil Java class, or the command-line "encrypt.bat" or "encrypt.sh" scripts if those are available to you. In order for the passwor...
• EventListener: Listens for and responds to crawler lifecycle events.
• Host: A host name and port configuration used for proxy or connection settings.
• Properties: A collection of key-value metadata properties for a document.
• PropertyMatcher: Match one or more document metadata field names and/or values.
• PropertyMatchers: A collection of property matchers for evaluating document metadata.
• ProxySettings: Proxy server connection settings including host, credentials, and scheme.
• Regex: A regular expression pattern with optional matching flags.
• RegexFieldValueExtractor: Extracts field names and values from text using a regular expression with capture groups.
• TextMatcher: A configurable class offering a few different ways to perform text matching and replacing. Note the replaceAll property is only applicable where replacement is supported.

Enumerations

• PropertySetter: Supported strategies for setting a metadata field value.