Package com.norconex.crawler.fs.fetch.impl.hdfs

Class HdfsFetcher

java.lang.Object

com.norconex.crawler.core.fetch.AbstractFetcher<HdfsFetcherConfig>

com.norconex.crawler.fs.fetch.impl.AbstractNioFetcher<HdfsFetcherConfig>

com.norconex.crawler.fs.fetch.impl.hdfs.HdfsFetcher

All Implemented Interfaces:

Configurable<HdfsFetcherConfig>, EventListener<Event>, Fetcher, EventListener, Consumer<Event>

public class HdfsFetcher
extends AbstractNioFetcher<HdfsFetcherConfig>

Fetcher for HDFS (webhdfs://), backed by a custom, read-only NIO.2 FileSystemProvider (HdfsFileSystemProvider) that speaks the WebHDFS REST API directly over Apache HttpClient - no Hadoop client dependency.

The reference host and port must be the NameNode's WebHDFS/HTTP port (typically 9870 on Hadoop 3.x), not the RPC port (8020) used by the native Hadoop client.

Field Summary

Fields inherited from interface com.norconex.commons.lang.config.Configurable

PROPERTY

Constructor Summary

Constructors

Constructor	Description
HdfsFetcher()

Method Summary

Modifier and Type	Method	Description
protected boolean	acceptRequest(@NonNull FetchRequest fetchRequest)	Optional method for subclasses to override when in need for additional acceptance logic beyond reference filters (which this abstract class provides).
protected boolean	canEqual(Object other)
boolean	equals(Object o)
protected void	fetcherShutdown(CrawlerSession crawler)	Invoked once per fetcher when the crawler ends.
protected void	fetcherStartup(CrawlerSession crawler)	Invoked once per fetcher instance, when the crawler starts.
HdfsFetcherConfig	getConfiguration()
int	hashCode()
protected Path	resolvePath(String reference)	Resolves a crawler reference (as found on Doc.getReference()) into a NIO.2 Path.
String	toString()

Methods inherited from class com.norconex.crawler.fs.fetch.impl.AbstractNioFetcher

fetch, fetchChildPaths, fetchContent, fetchFileObject, fetchMetadata, getOrOpenFileSystem

Methods inherited from class com.norconex.crawler.core.fetch.AbstractFetcher

accept, accept

Methods inherited from class java.lang.Object

clone, finalize, getClass, notify, notifyAll, wait, wait, wait

Methods inherited from interface java.util.function.Consumer

andThen

Constructor Details

HdfsFetcher

public HdfsFetcher()

Method Details

fetcherStartup

protected void fetcherStartup(CrawlerSession crawler)

Description copied from class: AbstractFetcher

Invoked once per fetcher instance, when the crawler starts. Default implementation does nothing.

Overrides:

fetcherStartup in class AbstractFetcher<HdfsFetcherConfig>

Parameters:

crawler - crawler

fetcherShutdown

protected void fetcherShutdown(CrawlerSession crawler)

Description copied from class: AbstractFetcher

Invoked once per fetcher when the crawler ends. Default implementation does nothing.

Overrides:

fetcherShutdown in class AbstractNioFetcher<HdfsFetcherConfig>

Parameters:

crawler - crawler

acceptRequest

protected boolean acceptRequest(@NonNull
 @NonNull FetchRequest fetchRequest)

Description copied from class: AbstractFetcher

Optional method for subclasses to override when in need for additional acceptance logic beyond reference filters (which this abstract class provides). This method is only invoked if there are no reference filters configured on this class, or all of them accepts the request document. Default implementation does nothing (always return true).

Overrides:

acceptRequest in class AbstractFetcher<HdfsFetcherConfig>

Parameters:

fetchRequest - the fetch request to evaluate (never null)

Returns:

true if accepted

resolvePath

protected Path resolvePath(String reference)
                    throws IOException

Description copied from class: AbstractNioFetcher

Resolves a crawler reference (as found on Doc.getReference()) into a NIO.2 Path.

Specified by:

resolvePath in class AbstractNioFetcher<HdfsFetcherConfig>

Parameters:

reference - the reference to resolve

Returns:

the resolved path

Throws:

IOException - if the reference cannot be resolved

toString

public String toString()

Overrides:

toString in class AbstractNioFetcher<HdfsFetcherConfig>

equals

public boolean equals(Object o)

Overrides:

equals in class AbstractNioFetcher<HdfsFetcherConfig>

canEqual

protected boolean canEqual(Object other)

Overrides:

canEqual in class AbstractNioFetcher<HdfsFetcherConfig>

hashCode

public int hashCode()

Overrides:

hashCode in class AbstractNioFetcher<HdfsFetcherConfig>

getConfiguration

public HdfsFetcherConfig getConfiguration()