Package com.norconex.crawler.fs.fetch.impl

Class AbstractNioFetcher<C extends BaseFetcherConfig>

java.lang.Object

com.norconex.crawler.core.fetch.AbstractFetcher<C>

com.norconex.crawler.fs.fetch.impl.AbstractNioFetcher<C>

Type Parameters:

C - configuration type

All Implemented Interfaces:

Configurable<C>, EventListener<Event>, Fetcher, EventListener, Consumer<Event>

Direct Known Subclasses:

AdlsGen2Fetcher, ArchiveFetcher, AzureBlobFetcher, CmisFetcher, FtpFetcher, GcsFetcher, HdfsFetcher, LocalFetcher, S3Fetcher, SftpFetcher, SmbFetcher, WebDavFetcher

public abstract class AbstractNioFetcher<C extends BaseFetcherConfig>
extends AbstractFetcher<C>

Base class for fetchers relying on Java NIO.2 (java.nio.file), either via the JDK's built-in providers or third-party FileSystemProvider implementations found on the classpath.

Field Summary

Fields inherited from interface com.norconex.commons.lang.config.Configurable

PROPERTY

Constructor Summary

Constructors

Constructor	Description
AbstractNioFetcher()

Method Summary

Modifier and Type	Method	Description
protected boolean	canEqual(Object other)
boolean	equals(Object o)
FetchResponse	fetch(FetchRequest fetchRequest)
protected FolderPathsFetchResponse	fetchChildPaths(FolderPathsFetchRequest req)
protected boolean	fetchContent(Doc doc, @NonNull Path path)
protected void	fetcherShutdown(CrawlerSession crawler)	Invoked once per fetcher when the crawler ends.
protected FileFetchResponse	fetchFileObject(FileFetchRequest req)
protected void	fetchMetadata(Doc doc, @NonNull Path path, @NonNull BasicFileAttributes attrs)
protected FileSystem	getOrOpenFileSystem(URI referenceUri, Map<String,?> env)	Gets the FileSystem for the given URI's host and port, opening and caching a new one on first use.
int	hashCode()
protected abstract Path	resolvePath(String reference)	Resolves a crawler reference (as found on Doc.getReference()) into a NIO.2 Path.
String	toString()

Methods inherited from class com.norconex.crawler.core.fetch.AbstractFetcher

accept, accept, acceptRequest, fetcherStartup

Methods inherited from class java.lang.Object

clone, finalize, getClass, notify, notifyAll, wait, wait, wait

Methods inherited from interface com.norconex.commons.lang.config.Configurable

getConfiguration

Methods inherited from interface java.util.function.Consumer

andThen

Constructor Details

AbstractNioFetcher

public AbstractNioFetcher()

Method Details

fetcherShutdown

protected void fetcherShutdown(CrawlerSession crawler)

Description copied from class: AbstractFetcher

Invoked once per fetcher when the crawler ends. Default implementation does nothing.

Overrides:

fetcherShutdown in class AbstractFetcher<C extends BaseFetcherConfig>

Parameters:

crawler - crawler

fetch

public FetchResponse fetch(FetchRequest fetchRequest)
                    throws FetchException

Throws:

FetchException

resolvePath

protected abstract Path resolvePath(String reference)
                             throws IOException

Resolves a crawler reference (as found on Doc.getReference()) into a NIO.2 Path.

Parameters:

reference - the reference to resolve

Returns:

the resolved path

Throws:

IOException - if the reference cannot be resolved

getOrOpenFileSystem

protected FileSystem getOrOpenFileSystem(URI referenceUri,
 Map<String,?> env)
                                  throws IOException

Gets the FileSystem for the given URI's host and port, opening and caching a new one on first use. Credentials always come from the supplied environment, never from the URI: some providers (e.g., sftp-fs/ftp-fs) echo the connected username back into the user-info component of paths obtained from a listing (via Path.toUri()), yet reject that same user-info component if it is fed back into FileSystems.newFileSystem(URI, Map). Any user-info on the given URI is therefore ignored.

Parameters:

referenceUri - a URI identifying the target file system (only its scheme, host and port are used)

env - environment used only the first time a given host/port is resolved

Returns:

the file system for that host and port

Throws:

IOException - if the file system could not be opened

fetchFileObject

protected FileFetchResponse fetchFileObject(FileFetchRequest req)
                                     throws FetchException

Throws:

FetchException

fetchChildPaths

protected FolderPathsFetchResponse fetchChildPaths(FolderPathsFetchRequest req)
                                            throws FetchException

Throws:

FetchException

fetchMetadata

protected void fetchMetadata(Doc doc,
 @NonNull
 @NonNull Path path,
 @NonNull
 @NonNull BasicFileAttributes attrs)
                      throws IOException

Throws:

IOException

fetchContent

protected boolean fetchContent(Doc doc,
 @NonNull
 @NonNull Path path)
                        throws IOException

Throws:

IOException

equals

public boolean equals(Object o)

Overrides:

equals in class AbstractFetcher<C extends BaseFetcherConfig>

canEqual

protected boolean canEqual(Object other)

Overrides:

canEqual in class AbstractFetcher<C extends BaseFetcherConfig>

hashCode

public int hashCode()

Overrides:

hashCode in class AbstractFetcher<C extends BaseFetcherConfig>

toString

public String toString()

Overrides:

toString in class AbstractFetcher<C extends BaseFetcherConfig>