Package com.norconex.crawler.core.doc.pipelines

Class DedupService

java.lang.Object

com.norconex.crawler.core.doc.pipelines.DedupService

public class DedupService
extends Object

Constructor Summary

Constructors

Constructor	Description
DedupService()

Method Summary

Modifier and Type	Method	Description
Optional<String>	findOrTrackDocument(CrawlerEntry crawlEntry)	Finds a document with the same checksum as the one supplied (effectively being a duplicate) and return its reference.
Optional<String>	findOrTrackMetadata(CrawlerEntry crawlEntry)	Finds a document with the same checksum as the one supplied (effectively being a duplicate) and return its reference.
void	init(CrawlerSession session)

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Details

DedupService

public DedupService()

Method Details

init

public void init(CrawlerSession session)

findOrTrackDocument

public Optional<String> findOrTrackDocument(CrawlerEntry crawlEntry)

Finds a document with the same checksum as the one supplied (effectively being a duplicate) and return its reference. Otherwise save the checksum of the supplied document and return an empty optional.

Parameters:

crawlEntry - doc to check for duplicates

Returns:

the duplicate reference or empty

findOrTrackMetadata

public Optional<String> findOrTrackMetadata(CrawlerEntry crawlEntry)

Finds a document with the same checksum as the one supplied (effectively being a duplicate) and return its reference. Otherwise save the checksum of the supplied document and return an empty optional.

Parameters:

crawlEntry - doc to check for duplicates

Returns:

the duplicate reference or empty