GenericRecrawlableResolver

Determines whether a previously crawled document should be re-crawled.

crawler/webJavadoc: GenericRecrawlableResolver

Notes

Determines whether a previously crawled document should be re-crawled on the current run, based on minimum recrawl frequency rules per content type or reference pattern, with optional sitemap directive support.

Examples

The following example ensures PDFs are re-crawled no more frequently than once a month, while HTML news can be re-crawled frequently at every half hour. For the rest, it relies on the website sitemap directives (if any).

YAML

class: GenericRecrawlableResolver
sitemapSupport: LAST
minFrequencies:
  - applyTo: CONTENT_TYPE
    value: monthly
    matcher:
      pattern: application/pdf
  - applyTo: REFERENCE
    value: "1800000"
    matcher:
      method: REGEX
      pattern: .*latest-news.*\.html

JSON

{
  "class": "GenericRecrawlableResolver",
  "sitemapSupport": "LAST",
  "minFrequencies": [
    {
      "applyTo": "CONTENT_TYPE",
      "value": "monthly",
      "matcher": {
        "pattern": "application/pdf"
      }
    },
    {
      "applyTo": "REFERENCE",
      "value": "1800000",
      "matcher": {
        "method": "REGEX",
        "pattern": ".*latest-news.*\\.html"
      }
    }
  ]
}

XML

<class>GenericRecrawlableResolver</class>
<sitemapSupport>LAST</sitemapSupport>
<minFrequencies>
  <minFrequency>
    <applyTo>CONTENT_TYPE</applyTo>
    <value>monthly</value>
    <matcher>
      <pattern>application/pdf</pattern>
    </matcher>
  </minFrequency>
  <minFrequency>
    <applyTo>REFERENCE</applyTo>
    <value>1800000</value>
    <matcher>
      <method>REGEX</method>
      <pattern>.*latest-news.*.html</pattern>
    </matcher>
  </minFrequency>
</minFrequencies>

Usage

Full configuration skeleton, with every option and its default

YAML

class: GenericRecrawlableResolver
minFrequencies:
  - applyTo: CONTENT_TYPE
    matcher: {}
    value: string
sitemapSupport: FIRST

JSON

{
  "class": "GenericRecrawlableResolver",
  "minFrequencies": [
    {
      "applyTo": "CONTENT_TYPE",
      "matcher": {},
      "value": "string"
    }
  ],
  "sitemapSupport": "FIRST"
}

XML

<recrawlableResolver>
  <class>GenericRecrawlableResolver</class>
  <minFrequencies>
    <minFrequency>
      <applyTo>CONTENT_TYPE</applyTo>
      <matcher/>
      <value>string</value>
    </minFrequency>
  </minFrequencies>
  <sitemapSupport>FIRST</sitemapSupport>
</recrawlableResolver>

Properties

Property	Type	Required	Default
minFrequencies	MinFrequency[]	No	-
sitemapSupport	enum	No	FIRST

Property Details

minFrequencies

A list of minimum recrawl frequency rules. Each rule specifies a frequency value (e.g., hourly, daily, monthly, or milliseconds) and a matcher applied to either the document reference or content type to determine when a document is eligible to be re-crawled.

sitemapSupport

Controls how sitemap recrawl directives (frequency and last-modified) interact with the configured minimum frequency rules. Supported values are FIRST (sitemap takes precedence, default), LAST (minimum frequency rules take precedence), and NEVER (sitemap directives are ignored).

Allowed Values

• FIRST
• LAST
• NEVER