Package com.norconex.crawler.core.doc.operations.checksum

Interface MetadataChecksummer

All Known Implementing Classes:
AbstractMetadataChecksummer, FsMetadataChecksummer, GenericMetadataChecksummer, LastModifiedMetadataChecksummer
public interface MetadataChecksummer

Creates a checksum representing a document based on document metadata values obtained prior to fetching that document (e.g. HTTP header values form an HTTP HEAD call, file properties, etc.). Checksums are used to quickly filter out documents that have already been processed or that have changed since a previous run.

Two or more Doc can hold different values, but be deemed logically the same. Such documents do not have to be equal, but they should return the same checksum. An example of this can be two different URLs pointing to the same document, where only a single instance should be kept.

There are no strict rules that define what is equivalent or not.

See Also:

  • Method Details

    • createMetadataChecksum

      String createMetadataChecksum(Properties metadata)
      Creates a metadata checksum.
      Parameters:
      metadata - all metadata values
      Returns:
      a checksum value