LastModifiedMetadataChecksummer

Uses the exact value of the "Last-Modified" HTTP header field.

crawler/webJavadoc: LastModifiedMetadataChecksummer

Notes

Generates a metadata checksum from the value of the HTTP Last-Modified response header. When the header value is unchanged from the previous crawl, the document is considered unmodified and may be skipped.

Examples

The following example will store the last modified date used for checksum purposes in a field called "metaChecksum".

YAML

class: LastModifiedMetadataChecksummer
keep: true
toField: metaChecksum

JSON

{
  "class": "LastModifiedMetadataChecksummer",
  "keep": true,
  "toField": "metaChecksum"
}

XML

<class>LastModifiedMetadataChecksummer</class>
<keep>true</keep>
<toField>metaChecksum</toField>

Usage

Full configuration skeleton, with every option and its default

YAML

class: LastModifiedMetadataChecksummer
keep: false
onSet: APPEND
toField: string

JSON

{
  "class": "LastModifiedMetadataChecksummer",
  "keep": false,
  "onSet": "APPEND",
  "toField": "string"
}

XML

<metadataChecksummer>
  <class>LastModifiedMetadataChecksummer</class>
  <keep>false</keep>
  <onSet>APPEND</onSet>
  <toField>string</toField>
</metadataChecksummer>

Properties

Property	Type	Required	Default
keep	boolean	No	-
onSet	enum	No	-
toField	string	No	-

Property Details

keep

Add the checksum to the document metadata.

onSet

What to do if a target field of the same name already exist.

Allowed Values

• APPEND
• PREPEND
• REPLACE
• OPTIONAL

toField

A target field name used to store the value.