All Classes and Interfaces
Class
Description
A base implementation for doing batch commits.
A base implementation taking care of basic plumbing, such as
firing main Committer events (including exceptions),
storing the Committer context (available via
AbstractCommitter.getCommitterContext()),
and adding support for filtering unwanted requests.Convenience class to encapsulate various delay strategies.
Base implementation for creating voluntary delays between URL downloads.
Abstract implementation of
DocumentChecksummer giving the option
to keep the generated checksum in a metadata field.
Base class implementing the
AbstractFetcher.accept(FetchRequest) method
using reference filters to determine if this fetcher will accept to fetch
a document, in addition to whatever logic implementing classes may provide
by optionally overriding AbstractFetcher.acceptRequest(FetchRequest)
(which otherwise always return true).
Base class for Committers writing to the local file system.
Abstract implementation of
MetadataChecksummer giving the option
to keep the generated checksum.
Base class for fetchers relying on Java NIO.2 (
java.nio.file),
either via the JDK's built-in providers or third-party
FileSystemProvider implementations found on the
classpath.
Base class for screenshot handlers that capture browser page images during
crawling.
Fetcher for Azure Data Lake Storage Gen2, using references such as
abfss://filesystem@account.dfs.core.windows.net/path/to/file.
Configuration for
AdlsGen2Fetcher.Holds all responses obtained from fetching a document
using one or multiple fetchers.
File System multi-response information obtained from fetching a document.
File System multi-response information obtained from fetching a document.
HTTP Multi-response information obtained from fetching a document.
Commits documents to Amazon CloudSearch.
Commits documents to Amazon CloudSearch.
For auto registering in
BeanMapper.Utility methods for fetcher implementations using Apache HttpClient.
Commits documents to Kafka via it's Producer API
Configuration for
ApacheKafkaCommitter.
For auto registering in
BeanMapper.This class is used by each crawler instance to capture the closest
redirect target whether it is part of a redirect chain or not.
Fetcher for archive file systems treated as virtual directory trees.
Configuration for
ArchiveFetcher.
Fetcher for Azure Blob Storage, using references such as
azblob://account/container/path/to/blob.
Configuration for
AzureBlobFetcher.
Commits documents to Microsoft Azure Search.
Azure Search Committer configuration.
For auto registering in
BeanMapper.
Extension of
AbstractNioFetcher, adding authentication support.Abstract implementation of
MetadataChecksummer giving the option
to keep the generated checksum.
A base implementation taking care of basic plumbing for field mappings
and restrictions for filtering unwanted requests.
Shared base class for condition configuration beans that support an optional
user-supplied label.
Base implementation for creating voluntary delays between URL downloads.
Shared base class for document handler configuration beans that support an
optional user-supplied label.
Base class implementing the
AbstractFetcher.accept(FetchRequest)
method using reference filters to determine if this fetcher will accept to
fetch a document, in addition to whatever logic implementing classes may
provide by optionally overriding
AbstractFetcher.acceptRequest(FetchRequest)
(which otherwise always return true).
Base class for Committers writing to the local file system.
Shared base class for importer configuration beans that support an optional
user-supplied label.
Functional interface for processing requests in batch (for committers
supporting it).
File system crawler-specific initialization before the crawler starts.
Captures the document's original, unparsed content and stores it,
Base64-encoded, in a metadata field.
BinaryContentTransformer configuration.
A condition based on whether the document content (default) or
any of the specified metadata fields are blank or inexistent.
A condition based on whether the document content (default) or
any of the specified metadata fields are blank or inexistent.
Box API fetcher.
Configuration for
BoxFetcher.Buffer related utility methods.
Adapter that wraps a
CacheEntryChangeListener and implements
Hazelcast's entry listener interfaces.Canonical names for distributed cache maps used by the crawler cluster.
Detects and return any canonical URL found in documents, whether from
the HTTP headers (metadata), or from a page content (usually HTML).
Unless we are ignoring canonical URL support, checks that a document is
canonical or reject it.
Changes the character case of a document content or matching fields
values according to one of the following methods:
Changes the character case of matching fields and values according to
one of the following methods:
Character set detector.
Transforms a document content (if needed) from a source character
encoding (charset) to a target one.
Transforms a document content (if needed) from a source character
encoding (charset) to a target one.
Character set utility methods.
Checksum stage utility methods.
Checksum utility methods.
Supplies chunks of text to the provided consumer.
Utilities when dealing with
DocHandlerContext.Base class for sub-commands.
Clean the Collector crawling history.
Validate configuration file format and quit.
Resolve all includes and variables substitution and print the
resulting configuration to facilitate sharing.
Launches a crawler command with given crawler specifications and
command-line arguments.
Runtime exception for command line errors.
Main entry point to the crawler command-line usage.
Start a crawler.
Stop a crawler.
Export crawl store to specified file.
Import crawl store from specified file.
Provides an HTTP admin interface for cluster management.
Runtime exception for cache-related errors.
CMIS-enabled Content Management Systems (CMS) fetcher
(Atom end-point), backed by a custom, read-only NIO.2
FileSystemProvider (CmisFileSystemProvider).
Configuration for
CmisFetcher.Attributes of a CMIS object, backed by the Atom "entry" XML document
used to derive them.
Reduces specified consecutive characters or strings to only one
instance (document content only).
Reduces specified consecutive characters or strings to only one
instance (document content only).
A crawler command launched within an active crawl session.
Common pipeline stage for committing documents.
Commits documents to their final destination (e.g. search engine).
Holds objects defined outside a committer but useful or required for the
committer execution.
For auto registering in
BeanMapper.Default committer events.
Triggered when something went wrong with committing.
A context object for crawler pipelines dealing
with
Committer.A committer queue, accumulating requests before they are committed
in one or multiple batches.
Triggered when something went wrong with the committer queue.
A document (or record, entry, entity, etc.) to be committed.
Wrapper around multiple Committers so they can all be handled as one
and facilitating integration with various clients.
Default committer events.
Triggered when an operation when wrong on one or more committers
registered to a
CommitterService
Committer-related utility methods.
Generic pipeline stage for detecting common document attributes, as
described in
CommonAttributesResolver.
Detects common document attributes and set them as document metadata fields.
Commonly used
TextMatcher instances.Commonly encountered restrictions that can be applied to
Properties
instances.A condition usually used in flow creation when configuring
importer handlers.
Conditionally execute one or more doc handlers (possibly including other
conditional doc handlers) upon matching a condition, or not.
A condition backed by a configuration bean that supports an optional
user-supplied label.
A document handler backed by a configuration bean that supports an optional
user-supplied label.
Define and add constant values to documents.
Define and add constant values to documents.
Main class to detect all content types.
Listen for coordinator change affecting only this node.
Copies metadata fields.
Copies metadata fields.
Counts the number of matches of a given string (or string pattern) and
store the resulting value in a field in the specified "toField".
Counts the number of matches of a given string (or string pattern) and
store the resulting value in a field in the specified "toField".
Wrapper around an existing cluster string cache, offering generic attribute
setters and getters for convenience.
Optional callbacks invoked at specific times in the crawler life-cycle.
Convenience interface providing a default implementation of
execute that ignores all commands but
CrawlerCommand and
offers an overloaded accept method with a single argument.
Base Crawler configuration.
Defines how changes are discovered during a crawl run.
Crawler class holding state properties required for running commands and
tasks.
For auto registering in
BeanMapper.It is assumed there will be one instance of this class per crawler defined.
Crawl document processing context, encapsulating both the document and its
crawl state/history from the crawl entry ledger.
Metadata constants for common metadata field
names typically set by a collector crawler.
Holds minimal meta information and state necessary to the proper
(re)processing of a document in the context of a crawl.
Tracks document state and any other meta information required
for document processing.
Prepare the doc processing ledger for a new or incremental crawl.
A crawler event.
Thin wrapper over
EventManager for firing crawler events.Runtime exception for most issues thrown by a crawler.
For auto registering in
BeanMapper.Listener adapter for crawler life-cycle events (e.g., start,
init, stop, clean).
Interface used to access crawler metrics across the application.
Offers useful methods for registering and unregistering MXBean to JMX.
Whether we are executing a full or incremental crawler run.
Creator of a crawl pipeline, the principal flow of a crawler.
Main crawler task, getting references from the crawl queue, fetch them,
and process them until the queue is empty.
What to do with ALL current entries in the queue.
Logs useful information about the crawler execution progress.
Describes the state of the crawler when launched (new launch or resuming a
non completed/failed one).
Coordinates once-per-session and once-per-run task deduplication using the
distributed cache, and exposes the run-scoped attribute store.
Details on the current crawl run.
Establishes the current crawl run information specific to this run.
A crawl session, representing a crawl from begin to finish, which may
include zero to multiple "resumes" from stops, or failures.
Manages persistent crawl-state and session-scoped attributes stored in the
distributed session cache.
For auto registering in
BeanMapper.Identifies a column in comma-separated-value (CSV) content.
Commits documents to CSV files (Comma Separated Value).
Commits documents to CSV files (Comma Separated Value).
Split files with Coma-Separated values (or any other characters, like tab)
into one document per line.
Split files with Coma-Separated values (or any other characters, like tab)
into one document per line.
Adds the current computer UTC date to the specified
field.Adds the current computer UTC date to the specified
field.
A condition based on the date value(s) of matching
metadata fields given the supplied date format.
A condition based on the date value(s) of matching
metadata fields given the supplied date format.
Formats a date from any given format to a format of choice, as per the
formatting options found on
ZonedDateTimeParser with the addition
of the format "EPOCH", which represents the difference, measured in
milliseconds, between the date and midnight, January 1, 1970.Formats a date from any given format to a format of choice, as per the
formatting options found on
ZonedDateTimeParser with the addition
of the format "EPOCH", which represents the difference, measured in
milliseconds, between the date and midnight, January 1, 1970.Provides a date time value which may be the same on each invocation, or
change each time, based on implementation.
Creates suppliers of static or dynamic dates, from a formatted string.
Returns a supplier that dynamically generates a date-time, that once
generated, never changes for that same supplier.
Returns a supplier that dynamically generates a date-time, that
changes with every invocation.
Static date-time, supplying the value passed in constructor.
Immutable date-value matcher, supporting both date expressions and
standard operators.
A utility tagger to help with troubleshooting of document importing.
A utility tagger to help with troubleshooting of document importing.
Parser class when no other handlers are specified.
Parser class when no other handlers are specified.
Resolves and creates intentional "delays" to increase document download
time intervals.
Wait for configured or default delay to expire.
Provides the ability to send deletion requests to your configured
committer(s) whenever a reference is rejected, regardless whether it was
encountered in a previous crawling session or not.
Configuration for
DeleteRejectedEventListener.A committer deletion request.
Delete the metadata fields provided.
Delete the metadata fields provided.
A document being imported.
Document Handler are responsible for interacting with a document to
either parse it, transform it, decorate it, split it, translate it, filter
it, etc.
Exception thrown by handler classes upon encountering
issues.
Handles images associated with a document (which is different than a document
being itself an image).
Configuration for
DocImageHandler.Directory structure when storing images on disk.
Constants for common metadata field names typically associated
with a document and often set on
Doc.getMetadata().
Creates a checksum representing a a document.
Common pipeline stage for creating a document checksum.
Optional custom processing of a document just before or just after
document has been imported by the Importer module.
Provided both document checksum and deduplication are enabled,
verify there are no other documents with the same content checksum
encountered in the same crawl session.
Filter a document after the document content is fetched, downloaded,
or otherwise read or acquired.
Adds the document length (i.e., number of bytes) to
the specified
field.Adds the document length (i.e., number of bytes) to
the specified
field.
A condition using a Document Object Model (DOM) representation of an HTML,
XHTML, or XML document content to match an element, attribute or value.
A condition using a Document Object Model (DOM) representation of an HTML,
XHTML, or XML document content to match an element, attribute or value.
Extracts links from a Document Object Model (DOM) representation of an
HTML, XHTML, or XML document content based on values of matching
elements and attributes.
Configuration for
DomLinkExtractor.DOM operation details.
Splits HTML, XHTML, or XML document on elements matching a given
selector.
Splits HTML, XHTML, or XML document on elements matching a given
selector.
Extract the value of one or more elements or attributes into
a target field, or delete matching elements.
Extract the value of one or more elements or attributes into
a target field, or delete matching elements.
Utility methods related to JSoup/DOM manipulation.
Egnyte API fetcher.
Configuration for
EgnyteFetcher.
Commits documents to Elasticsearch.
For auto registering in
BeanMapper.
For documents containing embedded documents (e.g. zip files), the default
behavior of this treat them as a single document, merging all
embedded documents content and metadata into the parent document.
Filters a reference based on a comma-separated list of extensions.
Filters a reference based on a comma-separated list of extensions.
Document processor that extract the "main" image from HTML pages.
Configuration for
FeaturedImageResolver.Type of featured image storages.
Directory structure when storing images on disk.
Whether to perform metadata extraction on its own before (or instead)
performing document extraction as a whole (including metadata).
Indicates whether a given fetch directive should be required to execute,
optional, or disabled.
Fetch resources.
Small shared helpers used by multiple fetcher implementations.
Checked exception thrown upon encountering an error fetching
a document.
Fetch request.
Utility methods for fetching.
A utility tagger that reports in a CSV file the fields discovered
in a crawl session, captured at the point of your choice in the
importing process.
A utility tagger that reports in a CSV file the fields discovered
in a crawl session, captured at the point of your choice in the
importing process.
Fetches (i.e. download/copy for processing) a document and/or its
metadata (e.g., file properties) depending on supplied
FetchDirective.Resolves a group of filters, marking the group as accepted
(
true) if All following conditions are met:
All filters NOT implementing OnMatchFilter return
true.A request to get all direct paths within a folder.
Forces a metadata field to be single-value.
Forces a metadata field to be single-value.
Deprecated, for removal: This API element is subject to removal in a future version.
A path being crawled holding relevant crawl state information.
Default implementation of
MetadataChecksummer which by default
returns the combined values of FsDocMetadata.LAST_MODIFIED
and FsDocMetadata.FILE_SIZE, separated with an underscore
(e.g. "14125443181234_123").
File System queue.
File sytem queue utility methods.
Configuration for
FtpFetcher.
Fetcher for Google Cloud Storage (
gs://bucket/object),
backed by the official Google-maintained
google-cloud-nio provider.
Configuration for
GcsFetcher.Generic canonical link detector.
Configuration for
GenericCanonicalLinkDetector.
Default implementation for creating voluntary delays between URL downloads.
Configuration for
GenericDelayResolver.
Generic implementation of
MetadataChecksummer that uses
specified field names and their values to create a checksum.
Configuration for
GenericMetadataChecksummer.
Accepts or rejects a reference based on whether one or more
metadata field values are matching.
Accepts or rejects a reference based on whether one or more
metadata field values are matching.
Relies on both sitemap directives and custom instructions for
establishing the minimum frequency between each document recrawl.
Configuration for
GenericRecrawlableResolver.Provide redirect URLs by grabbing them from the HTTP Response
Location header value.Configuration for
GenericRedirectUrlProvider.
Filters URL based on a matching expression.
Filters URL based on a matching expression.
If there is a sitemap defined as a start reference for the same URL web site,
this locator is not used.
Configuration for
GenericSitemapLocator.
Generic implementation of
SpoiledReferenceStrategizer that
offers a simple mapping between the crawl state of references that have
turned "bad" and the strategy to adopt for each.
Configuration for
GenericSpoiledReferenceStrategizer.
Generic implementation of
WebUrlNormalizer that should satisfy
most URL normalization needs.
Configuration for
GenericUrlNormalizer.
By default a crawler will try to follow all links it discovers.
Configuration for
GenericUrlScopeResolver.
Commits documents to Google Cloud Search using the official Google API
client.
Google Cloud Search Committer configuration.
Declares which metadata field holds a parent item reference to use
for Google Cloud Search ACL inheritance.
How ACLs are inherited from a parent item.
Maps a metadata field to a Google Cloud Search ACL principal.
Target of an ACL mapping.
Maps crawler metadata fields to Google Cloud Search predefined
metadata fields.
Type of principal an ACL mapping resolves to.
Priority/consistency mode used for indexing and delete requests.
Declares the Google Cloud Search structured data value type to use
for a given metadata field sent as structured data.
Google Cloud Search structured data value type.
Format used to upload document content.
For auto registering in
BeanMapper.Google Drive API fetcher for Google Workspace My Drive and Shared Drives.
Configuration scaffold for a future
GoogleDriveFetcher
implementation.
Grobid REST service configuration.
Strategy interface for building a Hazelcast
Config programmatically.Context passed to
HazelcastConfigurer.buildConfig(HazelcastConfigurerContext) with all
framework-supplied information needed to build the Hazelcast configuration.Hazelcast IMap-based implementation of the Cache interface.
WebHDFS authentication methods.
Fetcher for HDFS (
webhdfs://), backed by a custom, read-only
NIO.2 FileSystemProvider
(HdfsFileSystemProvider) that speaks the WebHDFS REST API
directly over Apache HttpClient - no Hadoop client dependency.
Configuration for
HdfsFetcher.Given a separator, split a field string into multiple segments
representing each node of a hierarchical branch.
Given a separator, split a field string into multiple segments
representing each node of a hierarchical branch.
Class handling HSTS support for servers supporting it.
A memory efficient HTML link extractor.
Configuration for
HtmlLinkExtractor.
Generic HTTP Fetcher authentication configuration.
Authentication methods.
Default web crawler implementation of
Fetcher, based on
Apache HttpClient.Generic HTTP Fetcher configuration.
HTTP response information obtained from fetching a document.
Fetches (i.e. download for processing) a document and/or its metadata
(HTTP response headers) depending on supplied
HttpMethod.
Used to set and capture HTTP request/response headers, when enabled.
Configuration for
HttpSniffer.
Commits documents to IDOL Server/DIH or Connector
Framework Server (CFS).
IDOL Committer configuration.
For auto registering in
BeanMapper.
Thread-safe, in-memory LRU cache for featured images, keyed by
image URL.
Transforms an image using common image operations.
Transforms an image using common image operations.
Principal class responsible for importing documents.
Importer configuration.
An Importer event.
Exception thrown when an issue prevented the proper importation of a file.
Command line launcher of the Importer application.
A context object for crawler pipelines dealing
with
ImporterResponse.
For auto registering in
BeanMapper.An Importer request, unique for each document to be imported.
Processes an importer response to modify it or perform other actions
as required before it is returned.
RuntimeException thrown when a an issue prevented the proper importation of a
file.
Common pipeline stage for importing documents.
In-memory
CacheMap backed by a ConcurrentHashMap.In-memory
CacheQueue backed by a ConcurrentLinkedQueue.In-memory
CacheSet backed by a concurrent hash set.A generic Hazelcast Compact serializer that delegates to Jackson
(via
SerialUtil) for the actual serialization.Default
HazelcastConfigurer implementation that builds the Hazelcast
configuration programmatically using a JDBC data source for persistence.
Commits documents to JSON files.
Configuration for the
JsonFileCommitter.Keep only the metadata fields provided, delete all other ones.
Keep only the metadata fields provided, delete all other ones.
Configuration for Kerberos/SPNEGO authentication against a secured
WebHDFS endpoint.
Configuration for Kerberos and SPNEGO authentication.
An importer flow node or configuration bean that exposes an optional
user-supplied label.
Detects a document language based on Apache Tika language detection
capability.
Detects a document language based on Apache Tika language detection
capability.
Default implementation of
MetadataChecksummer for the
Norconex Web Crawler which simply
returns the exact value of the "Last-Modified" HTTP header field, or
null if not present.Represents a link extracted from a document.
Responsible for finding links in documents.
Extract URLs before sending to importer (because the importer may
strip some "valid" urls in producing normalized content).
Fetcher for a local file system, backed by the JDK's default
FileSystemProvider.
Configuration for
LocalFetcher.Single-node pipeline manager that executes steps sequentially on the
calling thread.
WARNING: Not intended for production use.
Microsoft 365 Graph fetcher for SharePoint Online and OneDrive.
Configuration for
M365GraphFetcher.Implementation of
DocumentChecksummer which
returns a MD5 checksum value of the extracted document content unless
one or more given source fields are specified, in which case the MD5
checksum value is constructed from those fields.Configuration for
Md5DocumentChecksummer.In-memory
CacheManager for the memory (non-clustered) mode.Lightweight single-node
Cluster implementation that requires
no Hazelcast or any external infrastructure.Connector that creates a lightweight
MemoryCluster with
in-memory caches and no distributed infrastructure.
WARNING: Not intended for production use.
Merge multiple metadata fields into a single one.
Merge multiple metadata fields into a single one.
Creates a checksum representing a document based on document metadata
values obtained prior to fetching that document (e.g.
Creates a checksum based on fetched metadata.
Provided both document checksum and deduplication are enabled,
verify there are no other documents with the same metadata checksum.
Supported Google Cloud Search item metadata targets.
Filter a reference based on the metadata that could be obtained for a
document, before it was fetched, downloaded, or otherwise read or acquired
(e.g.
Filters documents based on fetched metadata.
Implementation of
Fetcher that groups multiple other fetcher
to treat them as a single fetcher.CacheManager backed by H2 MVStore for persistent caches and
in-memory structures for ephemeral caches (eph-*).File-backed
CacheMap implementation using H2 MVStore.File-backed FIFO
CacheQueue implementation using H2 MVStore.File-backed
CacheSet implementation using H2 MVStore.File-backed single-node
Cluster implementation using H2 MVStore.Connector that creates a file-backed
MVStoreCluster using
H2's MVStore engine.Configuration for the MVStore-based cluster connector.
Commit documents/fields to a Neo4j graph database.
Neo4j Committer configuration.
For auto registering in
BeanMapper.
A condition based on the numeric value(s) of matching
metadata fields, supporting decimals.
A condition based on the numeric value(s) of matching
metadata fields, supporting decimals.
OCR configuration details.
Constants indicating the action to perform upon matching a condition.
Tells the collector that a filter is of "OnMatch" type.
Runs the predicate against each list items and return
true
if all entries are accepted, or if there are items implementing
OnMatchFilter, if at least one of them is "included".
Act as a flag indicating if a document has been parsed or not in
a given process flow.
Split PDFs pages so each pages are treated as individual documents.
Split PDFs pages so each pages are treated as individual documents.
Runtime exception for cache-related errors.
Execute a worker and, if applicable, a coordinator to execute a pipeline.
Responsible for pipeline execution.
Snapshot representation a the pipeline progress.
MXBean implementation delegating to PipelineManager#getPipelineProgress.
Registrar for the coordinator-only PipelineProgress MXBean.
MXBean exposing cluster-wide pipeline progress (coordinator view).
Result summary of a pipeline execution.
For workers to know when a pipeline has terminated.
Playwright-supported browser engines.
Uses Microsoft Playwright
for browser-based crawling.
Configuration for
PlaywrightFetcher.Element locator strategy for
PlaywrightFetcherConfig.getWaitForElementSelector().
Takes screenshots of pages using a Playwright
Page.Queue for crawling URLs from matching metadata fields, if any.
Reference processing status.
The different stages of a crawl entry processing life-cycle:
UNTRACKED -> QUEUED -> PROCESSING -> PROCESSEDVery simple cache entry expression.
Holds contextual objects necessary to initialize a crawler queue.
Default queue initializer, feeding the queue from a mix of reference
enqueuers.
Hold necessary objects to a specific pipeline execution over a
CrawlerDocContext.Common pipeline stage for queuing documents.
Shared read-only
BasicFileAttributes behavior for remote file
systems that only differentiate between regular files and directories.Shared behavior for read-only NIO.2 file systems backed by remote
storage/providers.
Common read-only
FileSystemProvider behavior shared by cloud and
remote filesystem providers.Shared absolute-path behavior for remote, read-only file systems.
Indicates whether a document that was successfully crawled on a previous
crawling session should be recrawled or not.
Determines whether to recrawl a document or not.
Responsible for providing a target absolute URL each time an HTTP redirect
is encountered when invoking a URL.
A condition based on a text pattern matching a document reference (e.g.
A condition based on a text pattern matching a document reference (e.g.
Introduces different delays between document downloads based on matching
document reference (URL) patterns.
Configuration for
ReferenceDelayResolver.Provides initial references to be store in the queue for crawling.
Filter a document based on its reference, before its properties or content
gets read or otherwise acquired.
Common pipeline stage for filtering references.
Provide references for crawling.
Enqueues references from files obtained from the crawler configuration
CrawlerConfig.getStartReferencesFiles().Enqueues references from a list obtained from the crawler configuration
CrawlerConfig.getStartReferences().Enqueues references from files obtained from the crawler configuration
CrawlerConfig.getStartReferencesProviders().
Link extractor using regular expressions to extract links found in text
documents.
Configuration for
RegexLinkExtractor.
Extracts field names and their values with regular expression.
Extracts field names and their values with regular expression.
Special handler that marks a document as being "rejected", with an
optional custom message for event logging.
Rename metadata fields to different names.
Rename metadata fields to different names.
Replaces every occurrences of the given replacements
(document content only).
Replaces every occurrences of the given replacements
(document content only).
Queue orphans for deletion.
Queue orphans for processing.
Robots meata creation.
Robots Meta NoIndex Check.
Responsible for extracting robot information from a page.
Immutable holder of robots.txt rules.
Holds a robots.txt rule.
Apply robot rules provided by resolved RobotsTxt.
Given a URL, extract any "robots.txt" rules.
Fetcher for Amazon S3 (
s3://bucket/key), backed by a
custom, read-only NIO.2 FileSystemProvider
(S3FileSystemProvider) built directly on the standard AWS SDK
v2 S3Client - not the AWS Common Runtime (CRT) client, avoiding
its ~19MB native-library dependency.
Configuration for
S3Fetcher.
Saves a copy of the document at its current processing state in
the specified directory.
Saves a copy of the document at its current processing state in
the specified directory.
Creates
ThreadFactory instances with a thread name prefixed with
a scope name and can log extra information on threads at job run begin/stop
(if DEBUG is enabled).
Configuration for screenshot handlers (Selenium WebDriver and Playwright).
A condition formulated using a scripting language.
A condition formulated using a scripting language.
Runs scripts written in supported programming languages.
Transform incoming documents using a scripting language.
Transform incoming documents using a scripting language.
Filters URL based based on the number of URL segments.
Configuration for
SegmentCountUrlFilter.
Tika sentiment analysis configuration (via Tika's
SentimentAnalysisParser).Backend-agnostic envelope for storing arbitrary objects as a pair of:
- className: fully qualified class name of the original object
- serialized: JSON representation of the original object
This type intentionally has no backend-specific annotations so it can be
used in any storage path (e.g., String caches, DB) and adapted by
backend-specific wrappers (e.g., Protostream-annotated classes).
SFTP fetcher (
sftp://), backed by the
sftp-fs NIO.2
file system provider.
Configuration for
SftpFetcher.Sitemap change frequency unit, as defined on
http://www.sitemaps.org/protocol.html
Suggestions of sitemap locations associated with a URL web site.
CIFS fetcher (Samba, Windows share) (
smb://), backed by a
custom, read-only NIO.2 FileSystemProvider
(SmbFileSystemProvider) built directly on jcifs-ng's classic
SmbFile API.
Configuration for
SmbFetcher.Supported
SolrClient types.
Commits documents to Apache Solr.
Configuration for
SolrCommitter.
For auto registering in
BeanMapper.Splits an existing metadata value into multiple values based on a given
value separator (the separator gets discarded).
Splits an existing metadata value into multiple values based on a given
value separator (the separator gets discarded).
Decides which strategy to adopt for a given reference with a bad state.
Markers indicating what to do with references that were once processed
properly, but failed to get a good processing state a subsequent time around.
Commit documents to an SQL table.
SQL Committer configuration.
For auto registering in
BeanMapper.Implementation of
RobotsMetaProvider as per X-Robots-Tag
and ROBOTS standards.Configuration for
StandardRobotsMetaProvider.
Implementation of
RobotsTxtProvider as per the robots.txt standard
described at
http://www.robotstxt.org/robotstxt.html.A record holding information on a pipeline execution current step.
Hazelcast Compact serializer for
StepRecord.Command to stop a running crawler session.
Alternative to
CrawlerConfig.setMaxDocuments(int) for issuing a
crawler stop request upon reaching specific event counts.
Configuration for
StopCrawlerOnMaxEventListener.JDBC-based key-value MapStore for Hazelcast OSS.
JDBC-backed Hazelcast QueueStore implementation that persists
queue items in a text column in a table with an order column.
Sanitizes a string by stripping all characters but those identified.
Strips any content found after first match found for given pattern.
Strips any content found after first match found for given pattern.
Strips any content found before first match found for given pattern.
Strips any content found before first match found for given pattern.
Strips any content found between a matching start and end strings.
Strips any content found between a matching start and end strings.
Keep a substring of the content matching a begin and end character
indexes.
Keep a substring of the content matching a begin and end character
indexes.
Extracts and add values found between a matching start and
end strings to a document metadata field.
Extracts and add values found between a matching start and
end strings to a document metadata field.
A condition based on a text pattern matching a document content
(default), or matching specific field(s).
A condition based on a text pattern matching a document content
(default), or matching specific field(s).
Analyzes the content of the supplied document and adds statistical
information about its content or field as metadata fields.
Analyzes the content of the supplied document and adds statistical
information about its content or field as metadata fields.
Implementation of
LinkExtractor using
Apache Tika to perform URL
extractions from HTML documents.
Configuration for
TikaLinkExtractor.Raw Tika parser that needs to be configured by passing a tika
configuration file.
Attempts to generate a title from the document content (default) or
a specified metadata field.
Attempts to generate a title from the document content (default) or
a specified metadata field.
Translate documents using one of the supported translation API.
Translate documents using one of the supported translation API.
Truncates a
fromField value(s) and optionally replace truncated
portion by a hash value to help ensure uniqueness (not 100% guaranteed to
be collision-free).
Truncates a
fromField value(s) and optionally replace truncated
portion by a hash value to help ensure uniqueness (not 100% guaranteed to
be collision-free).A very unsafe trust manager accepting ALL certificates.
Factory that produces MapStore instances which serialize/deserialize
values to/from JSON strings and delegate persistence to
StringJdbcMapStore.Typed wrapper around StringJdbcQueueStore that serializes objects to
JSON strings when storing and deserializes JSON strings when loading.
Factory to create a Hazelcast JDBC-backed QueueStore.
A committer upsert request (update or insert).
Extracts unique URLs matching specific patterns in plain text content and
store them in a given field.
Extracts unique URLs matching specific patterns in plain text content and
store them in a given field.
Store on file all URLs that were "fetched", along with their HTTP response
code.
Configuration for
UrlStatusCrawlerEventListener.Generates a random Universally unique identifier (UUID) and stores it
in the specified
field.Generates a random Universally unique identifier (UUID) and stores it
in the specified
field.Facade for launching or obtaining a Web Crawler.
Web Crawler configuration, adding more options to the base
CrawlerConfig.Flags for storing as metadata a page referenced links.
A URL being crawled holding relevant crawl information.
HTTP Crawler event names.
Fetcher for WebDAV repositories (
webdav://,
webdavs://), backed by a custom, read-only NIO.2
FileSystemProvider
(WebDavFileSystemProvider) that speaks WebDAV directly over
Apache HttpClient (no third-party WebDAV library).
Configuration for
WebDavFetcher.Metadata constants for common metadata field
names typically set by the Web Crawler crawler.
A web browser.
Uses Selenium WebDriver support for using native browsers to crawl documents.
Configuration for
WebDriverFetcher.
Takes screenshot of pages using a Selenium
WebDriver.HTTP Fetch request.
HTTP response information obtained from fetching a document.
Represents a URL crawling status.
Responsible for normalizing URLs.
Parser for PureEdge Extensible Forms Description Language (XFDL).
Configuration for
XmlFeedLinkExtractor.
Commits documents to XML files.
Configuration for the
XmlFileCommitter.Splits XML document on a specific element.
Splits XML document on a specific element.
FsCrawlerEntryinstead.