Uses of Interface
com.norconex.crawler.core.fetch.Fetcher

Packages that use Fetcher

Package	Description
com.norconex.crawler.core
com.norconex.crawler.core.context
com.norconex.crawler.core.doc.operations
com.norconex.crawler.core.fetch
com.norconex.crawler.fs.fetch.impl
com.norconex.crawler.fs.fetch.impl.adlsgen2
com.norconex.crawler.fs.fetch.impl.archive
com.norconex.crawler.fs.fetch.impl.azureblob
com.norconex.crawler.fs.fetch.impl.box
com.norconex.crawler.fs.fetch.impl.cmis
com.norconex.crawler.fs.fetch.impl.egnyte
com.norconex.crawler.fs.fetch.impl.ftp
com.norconex.crawler.fs.fetch.impl.gcs
com.norconex.crawler.fs.fetch.impl.googledrive
com.norconex.crawler.fs.fetch.impl.hdfs
com.norconex.crawler.fs.fetch.impl.local
com.norconex.crawler.fs.fetch.impl.m365graph
com.norconex.crawler.fs.fetch.impl.s3
com.norconex.crawler.fs.fetch.impl.sftp
com.norconex.crawler.fs.fetch.impl.smb
com.norconex.crawler.fs.fetch.impl.webdav
com.norconex.crawler.web.doc.operations.image.impl
com.norconex.crawler.web.doc.operations.robot
com.norconex.crawler.web.doc.operations.robot.impl
com.norconex.crawler.web.doc.operations.sitemap
com.norconex.crawler.web.fetch.impl.httpclient
com.norconex.crawler.web.fetch.impl.playwright
com.norconex.crawler.web.fetch.impl.webdriver

Uses of Fetcher in com.norconex.crawler.core

Methods in com.norconex.crawler.core that return types with arguments of type Fetcher

Modifier and Type	Method	Description
List<Fetcher>	CrawlerConfig.getFetchers()	One or more fetchers responsible for pulling documents and document metadata associated with a reference from a source.

Method parameters in com.norconex.crawler.core with type arguments of type Fetcher

Modifier and Type	Method	Description
CrawlerConfig	CrawlerConfig.setFetchers(List<Fetcher> fetchers)	One or more fetchers responsible for pulling documents and document metadata associated with a reference from a source.

Uses of Fetcher in com.norconex.crawler.core.context

Methods in com.norconex.crawler.core.context that return Fetcher

Modifier and Type	Method	Description
Fetcher	CrawlerContext.getFetcher()

Methods in com.norconex.crawler.core.context with parameters of type Fetcher

Modifier and Type	Method	Description
CrawlerContext.Builder	CrawlerContext.Builder.fetcher(Fetcher fetcher)

Uses of Fetcher in com.norconex.crawler.core.doc.operations

Methods in com.norconex.crawler.core.doc.operations with parameters of type Fetcher

Modifier and Type	Method	Description
void	DocumentConsumer.accept(Fetcher fetcher, Doc doc)	Processes a document.

Uses of Fetcher in com.norconex.crawler.core.fetch

Classes in com.norconex.crawler.core.fetch that implement Fetcher

Modifier and Type	Class	Description
class	AbstractFetcher<C extends BaseFetcherConfig>	Base class implementing the AbstractFetcher.accept(FetchRequest) method using reference filters to determine if this fetcher will accept to fetch a document, in addition to whatever logic implementing classes may provide by optionally overriding AbstractFetcher.acceptRequest(FetchRequest) (which otherwise always return true).
class	MultiFetcher	Implementation of Fetcher that groups multiple other fetcher to treat them as a single fetcher.

Methods in com.norconex.crawler.core.fetch that return types with arguments of type Fetcher

Modifier and Type	Method	Description
List<Fetcher>	MultiFetcher.getFetchers()

Method parameters in com.norconex.crawler.core.fetch with type arguments of type Fetcher

Modifier and Type	Method	Description
MultiFetcher.Builder	MultiFetcher.Builder.fetchers(List<? extends Fetcher> fetchers)

Uses of Fetcher in com.norconex.crawler.fs.fetch.impl

Classes in com.norconex.crawler.fs.fetch.impl that implement Fetcher

Modifier and Type	Class	Description
class	AbstractNioFetcher<C extends BaseFetcherConfig>	Base class for fetchers relying on Java NIO.2 (java.nio.file), either via the JDK's built-in providers or third-party FileSystemProvider implementations found on the classpath.

Uses of Fetcher in com.norconex.crawler.fs.fetch.impl.adlsgen2

Classes in com.norconex.crawler.fs.fetch.impl.adlsgen2 that implement Fetcher

Modifier and Type	Class	Description
class	AdlsGen2Fetcher	Fetcher for Azure Data Lake Storage Gen2, using references such as abfss://filesystem@account.dfs.core.windows.net/path/to/file.

Uses of Fetcher in com.norconex.crawler.fs.fetch.impl.archive

Classes in com.norconex.crawler.fs.fetch.impl.archive that implement Fetcher

Modifier and Type	Class	Description
class	ArchiveFetcher	Fetcher for archive file systems treated as virtual directory trees.

Uses of Fetcher in com.norconex.crawler.fs.fetch.impl.azureblob

Classes in com.norconex.crawler.fs.fetch.impl.azureblob that implement Fetcher

Modifier and Type	Class	Description
class	AzureBlobFetcher	Fetcher for Azure Blob Storage, using references such as azblob://account/container/path/to/blob.

Uses of Fetcher in com.norconex.crawler.fs.fetch.impl.box

Classes in com.norconex.crawler.fs.fetch.impl.box that implement Fetcher

Modifier and Type	Class	Description
class	BoxFetcher	Box API fetcher.

Uses of Fetcher in com.norconex.crawler.fs.fetch.impl.cmis

Classes in com.norconex.crawler.fs.fetch.impl.cmis that implement Fetcher

Modifier and Type	Class	Description
class	CmisFetcher	CMIS-enabled Content Management Systems (CMS) fetcher (Atom end-point), backed by a custom, read-only NIO.2 FileSystemProvider (CmisFileSystemProvider).

Uses of Fetcher in com.norconex.crawler.fs.fetch.impl.egnyte

Classes in com.norconex.crawler.fs.fetch.impl.egnyte that implement Fetcher

Modifier and Type	Class	Description
class	EgnyteFetcher	Egnyte API fetcher.

Uses of Fetcher in com.norconex.crawler.fs.fetch.impl.ftp

Classes in com.norconex.crawler.fs.fetch.impl.ftp that implement Fetcher

Modifier and Type	Class	Description
class	FtpFetcher	FTP (ftp://) and FTPS (ftps://) fetcher, backed by the ftp-fs NIO.2 file system provider.

Uses of Fetcher in com.norconex.crawler.fs.fetch.impl.gcs

Classes in com.norconex.crawler.fs.fetch.impl.gcs that implement Fetcher

Modifier and Type	Class	Description
class	GcsFetcher	Fetcher for Google Cloud Storage (gs://bucket/object), backed by the official Google-maintained google-cloud-nio provider.

Uses of Fetcher in com.norconex.crawler.fs.fetch.impl.googledrive

Classes in com.norconex.crawler.fs.fetch.impl.googledrive that implement Fetcher

Modifier and Type	Class	Description
class	GoogleDriveFetcher	Google Drive API fetcher for Google Workspace My Drive and Shared Drives.

Uses of Fetcher in com.norconex.crawler.fs.fetch.impl.hdfs

Classes in com.norconex.crawler.fs.fetch.impl.hdfs that implement Fetcher

Modifier and Type	Class	Description
class	HdfsFetcher	Fetcher for HDFS (webhdfs://), backed by a custom, read-only NIO.2 FileSystemProvider (HdfsFileSystemProvider) that speaks the WebHDFS REST API directly over Apache HttpClient - no Hadoop client dependency.

Uses of Fetcher in com.norconex.crawler.fs.fetch.impl.local

Classes in com.norconex.crawler.fs.fetch.impl.local that implement Fetcher

Modifier and Type	Class	Description
class	LocalFetcher	Fetcher for a local file system, backed by the JDK's default FileSystemProvider.

Uses of Fetcher in com.norconex.crawler.fs.fetch.impl.m365graph

Classes in com.norconex.crawler.fs.fetch.impl.m365graph that implement Fetcher

Modifier and Type	Class	Description
class	M365GraphFetcher	Microsoft 365 Graph fetcher for SharePoint Online and OneDrive.

Uses of Fetcher in com.norconex.crawler.fs.fetch.impl.s3

Classes in com.norconex.crawler.fs.fetch.impl.s3 that implement Fetcher

Modifier and Type	Class	Description
class	S3Fetcher	Fetcher for Amazon S3 (s3://bucket/key), backed by a custom, read-only NIO.2 FileSystemProvider (S3FileSystemProvider) built directly on the standard AWS SDK v2 S3Client - not the AWS Common Runtime (CRT) client, avoiding its ~19MB native-library dependency.

Uses of Fetcher in com.norconex.crawler.fs.fetch.impl.sftp

Classes in com.norconex.crawler.fs.fetch.impl.sftp that implement Fetcher

Modifier and Type	Class	Description
class	SftpFetcher	SFTP fetcher (sftp://), backed by the sftp-fs NIO.2 file system provider.

Uses of Fetcher in com.norconex.crawler.fs.fetch.impl.smb

Classes in com.norconex.crawler.fs.fetch.impl.smb that implement Fetcher

Modifier and Type	Class	Description
class	SmbFetcher	CIFS fetcher (Samba, Windows share) (smb://), backed by a custom, read-only NIO.2 FileSystemProvider (SmbFileSystemProvider) built directly on jcifs-ng's classic SmbFile API.

Uses of Fetcher in com.norconex.crawler.fs.fetch.impl.webdav

Classes in com.norconex.crawler.fs.fetch.impl.webdav that implement Fetcher

Modifier and Type	Class	Description
class	WebDavFetcher	Fetcher for WebDAV repositories (webdav://, webdavs://), backed by a custom, read-only NIO.2 FileSystemProvider (WebDavFileSystemProvider) that speaks WebDAV directly over Apache HttpClient (no third-party WebDAV library).

Uses of Fetcher in com.norconex.crawler.web.doc.operations.image.impl

Methods in com.norconex.crawler.web.doc.operations.image.impl with parameters of type Fetcher

Modifier and Type	Method	Description
void	FeaturedImageResolver.accept(Fetcher fetcher, Doc doc)

Uses of Fetcher in com.norconex.crawler.web.doc.operations.robot

Methods in com.norconex.crawler.web.doc.operations.robot with parameters of type Fetcher

Modifier and Type	Method	Description
RobotsTxt	RobotsTxtProvider.getRobotsTxt(Fetcher fetchClient, String url)	Gets robots.txt rules.

Uses of Fetcher in com.norconex.crawler.web.doc.operations.robot.impl

Methods in com.norconex.crawler.web.doc.operations.robot.impl with parameters of type Fetcher

Modifier and Type	Method	Description
RobotsTxt	StandardRobotsTxtProvider.getRobotsTxt(Fetcher fetcher, String url)

Uses of Fetcher in com.norconex.crawler.web.doc.operations.sitemap

Methods in com.norconex.crawler.web.doc.operations.sitemap that return Fetcher

Modifier and Type	Method	Description
Fetcher	SitemapContext.getFetcher()

Methods in com.norconex.crawler.web.doc.operations.sitemap with parameters of type Fetcher

Modifier and Type	Method	Description
SitemapContext.SitemapContextBuilder	SitemapContext.SitemapContextBuilder.fetcher(Fetcher fetcher)

Uses of Fetcher in com.norconex.crawler.web.fetch.impl.httpclient

Classes in com.norconex.crawler.web.fetch.impl.httpclient that implement Fetcher

Modifier and Type	Class	Description
class	HttpClientFetcher	Default web crawler implementation of Fetcher, based on Apache HttpClient.

Uses of Fetcher in com.norconex.crawler.web.fetch.impl.playwright

Classes in com.norconex.crawler.web.fetch.impl.playwright that implement Fetcher

Modifier and Type	Class	Description
class	PlaywrightFetcher	Uses Microsoft Playwright for browser-based crawling.

Uses of Fetcher in com.norconex.crawler.web.fetch.impl.webdriver

Classes in com.norconex.crawler.web.fetch.impl.webdriver that implement Fetcher

Modifier and Type	Class	Description
class	WebDriverFetcher	Uses Selenium WebDriver support for using native browsers to crawl documents.