GenericSpoiledReferenceStrategizer

Generic strategy mapping between each references that have turned "bad" and their state.

crawler/coreJavadoc: GenericSpoiledReferenceStrategizer

Notes

Defines how the crawler handles documents that could not be retrieved ("spoiled" references), mapping each failure reason to a strategy such as DELETE, GRACE_ONCE, or IGNORE. A fallback strategy applies to any reason not explicitly mapped.

Examples

The following example represents the default configuration. It indicates that documents that are no longer found (e.g., 404) are to result in a Committer deletion request. Other causes for not being able to obtain a document will be graced once (re-attempted one last time on next run).

YAML

class: GenericSpoiledReferenceStrategizer
fallbackStrategy: DELETE
mappings:
  NOT_FOUND: DELETE
  BAD_STATUS: GRACE_ONCE
  ERROR: GRACE_ONCE

JSON

{
  "class": "GenericSpoiledReferenceStrategizer",
  "fallbackStrategy": "DELETE",
  "mappings": {
    "NOT_FOUND": "DELETE",
    "BAD_STATUS": "GRACE_ONCE",
    "ERROR": "GRACE_ONCE"
  }
}

XML

<class>GenericSpoiledReferenceStrategizer</class>
<fallbackStrategy>DELETE</fallbackStrategy>
<mappings>
  <mapping>
    <key>NOT_FOUND</key>
    <value>DELETE</value>
  </mapping>
  <mapping>
    <key>BAD_STATUS</key>
    <value>GRACE_ONCE</value>
  </mapping>
  <mapping>
    <key>ERROR</key>
    <value>GRACE_ONCE</value>
  </mapping>
</mappings>

Usage

Full configuration skeleton, with every option and its default

YAML

class: GenericSpoiledReferenceStrategizer
fallbackStrategy: DELETE
mappings: {}

JSON

{
  "class": "GenericSpoiledReferenceStrategizer",
  "fallbackStrategy": "DELETE",
  "mappings": {}
}

XML

<spoiledReferenceStrategizer>
  <class>GenericSpoiledReferenceStrategizer</class>
  <fallbackStrategy>DELETE</fallbackStrategy>
  <mappings/>
</spoiledReferenceStrategizer>

Properties

Property	Type	Required	Default
fallbackStrategy	enum	No	DELETE
mappings	Record<string, enum>	No	{"ERROR":GRACE_ONCE,"BAD_STATUS":GRACE_ONCE,"NOT_FOUND":DELETE}

Property Details

fallbackStrategy

The strategy to apply for any spoiled reference reason not explicitly mapped. Default is DELETE. Supported values are DELETE (send to committer for deletion), GRACE_ONCE (retry one more time on the next crawl), and IGNORE (do nothing).

Allowed Values

• DELETE
• GRACE_ONCE
• IGNORE

mappings

A map of processing outcomes to spoiled reference strategies. Supported outcomes include NOT_FOUND, BAD_STATUS, and ERROR. Each maps to a strategy of DELETE, GRACE_ONCE, or IGNORE. Default maps NOT_FOUND to DELETE and BAD_STATUS/ERROR to GRACE_ONCE.