Md5DocumentChecksummer

Generate a MD5 checksum from content (default) or fields.

crawler/coreJavadoc: Md5DocumentChecksummer

Notes

Generates an MD5 checksum from a document's content, metadata fields, or both to detect changes between crawl sessions. Optionally stores the generated checksum value in a specified metadata field.

Examples

The following example uses the document body (default) to create the checksum.

YAML

documentChecksummer:
  class: Md5DocumentChecksummer

JSON

{
  "documentChecksummer": {
    "class": "Md5DocumentChecksummer"
  }
}

XML

<documentChecksummer>
  <class>Md5DocumentChecksummer</class>
</documentChecksummer>

Usage

Full configuration skeleton, with every option and its default

YAML

class: Md5DocumentChecksummer
combineFieldsAndContent: false
fieldMatcher:
  ignoreCase: false
  ignoreDiacritic: false
  matchEmpty: false
  method: BASIC
  negateMatches: false
  partial: false
  pattern: string
  replaceAll: false
  trim: false
keep: false
onSet: APPEND
toField: string

JSON

{
  "class": "Md5DocumentChecksummer",
  "combineFieldsAndContent": false,
  "fieldMatcher": {
    "ignoreCase": false,
    "ignoreDiacritic": false,
    "matchEmpty": false,
    "method": "BASIC",
    "negateMatches": false,
    "partial": false,
    "pattern": "string",
    "replaceAll": false,
    "trim": false
  },
  "keep": false,
  "onSet": "APPEND",
  "toField": "string"
}

XML

<documentChecksummer>
  <class>Md5DocumentChecksummer</class>
  <combineFieldsAndContent>false</combineFieldsAndContent>
  <fieldMatcher>
    <ignoreCase>false</ignoreCase>
    <ignoreDiacritic>false</ignoreDiacritic>
    <matchEmpty>false</matchEmpty>
    <method>BASIC</method>
    <negateMatches>false</negateMatches>
    <partial>false</partial>
    <pattern>string</pattern>
    <replaceAll>false</replaceAll>
    <trim>false</trim>
  </fieldMatcher>
  <keep>false</keep>
  <onSet>APPEND</onSet>
  <toField>string</toField>
</documentChecksummer>

Properties

Property	Type	Required	Default
combineFieldsAndContent	boolean	No	false
fieldMatcher	TextMatcher	No	TextMatcher
keep	boolean	No	false
onSet	enum	No	-
toField	string	No	-

Property Details

combineFieldsAndContent

Combine content and matching field values to create the MD5 checksum. Otherwise, only content or fields will be used based on whether a field expression was used or not.

fieldMatcher

Use the values of matching fields to create the MD5 checksum. When set, it is used in lieu of the document content unless you opt to combine both.

keep

Add the checksum to the document metadata.

onSet

What to do if a target field of the same name already exist.

Allowed Values

• APPEND
• PREPEND
• REPLACE
• OPTIONAL

toField

A target field name used to store the value.