GenericMetadataChecksummer

Use one or more matching metadata field values.

crawler/coreJavadoc: GenericMetadataChecksummer

Notes

Generates a checksum from the values of one or more matching metadata fields to detect document changes between crawl sessions. When a document's metadata checksum matches its previous value, the document is considered unchanged.

Examples

The following example uses a combination of two (fictitious) fields called "docLastModified" and "docSize" to make the checksum.

YAML

metadataChecksummer:
  class: GenericMetadataChecksummer
  fieldMatcher:
    method: CSV
    pattern: docLastModified,docSize

JSON

{
  "metadataChecksummer": {
    "class": "GenericMetadataChecksummer",
    "fieldMatcher": {
      "method": "CSV",
      "pattern": "docLastModified,docSize"
    }
  }
}

XML

<metadataChecksummer>
  <class>GenericMetadataChecksummer</class>
  <fieldMatcher>
    <method>CSV</method>
    <pattern>docLastModified,docSize</pattern>
  </fieldMatcher>
</metadataChecksummer>

Usage

Full configuration skeleton, with every option and its default

YAML

class: GenericMetadataChecksummer
fieldMatcher:
  ignoreCase: false
  ignoreDiacritic: false
  matchEmpty: false
  method: BASIC
  negateMatches: false
  partial: false
  pattern: string
  replaceAll: false
  trim: false
keep: false
onSet: APPEND
toField: string

JSON

{
  "class": "GenericMetadataChecksummer",
  "fieldMatcher": {
    "ignoreCase": false,
    "ignoreDiacritic": false,
    "matchEmpty": false,
    "method": "BASIC",
    "negateMatches": false,
    "partial": false,
    "pattern": "string",
    "replaceAll": false,
    "trim": false
  },
  "keep": false,
  "onSet": "APPEND",
  "toField": "string"
}

XML

<metadataChecksummer>
  <class>GenericMetadataChecksummer</class>
  <fieldMatcher>
    <ignoreCase>false</ignoreCase>
    <ignoreDiacritic>false</ignoreDiacritic>
    <matchEmpty>false</matchEmpty>
    <method>BASIC</method>
    <negateMatches>false</negateMatches>
    <partial>false</partial>
    <pattern>string</pattern>
    <replaceAll>false</replaceAll>
    <trim>false</trim>
  </fieldMatcher>
  <keep>false</keep>
  <onSet>APPEND</onSet>
  <toField>string</toField>
</metadataChecksummer>

Properties

Property	Type	Required	Default
fieldMatcher	TextMatcher	No	TextMatcher
keep	boolean	No	false
onSet	enum	No	-
toField	string	No	-

Property Details

fieldMatcher

The values of matching fields will be used to create the checksum.

keep

Add the checksum to the document metadata.

onSet

What to do if a target field of the same name already exist.

Allowed Values

• APPEND
• PREPEND
• REPLACE
• OPTIONAL

toField

A target field name used to store the value.