ReferenceDelayResolver

Apply different delays per different URL patterns.

crawler/webJavadoc: ReferenceDelayResolver

Notes

Applies different crawl delays based on URL patterns. A default delay applies when no pattern matches, and specific URL patterns can override it with longer or shorter delays. Optionally ignores the crawl delay from robots.txt.

Examples

The following example will increase the delay to 10 seconds when encountering PDFs from a default of 3 seconds.

YAML

class: ReferenceDelayResolver
defaultDelay: 3000
delayReferencePatterns:
  - delay: 10000
    pattern: .*\.pdf

JSON

{
  "class": "ReferenceDelayResolver",
  "defaultDelay": 3000,
  "delayReferencePatterns": [
    {
      "delay": 10000,
      "pattern": ".*\\.pdf"
    }
  ]
}

XML

<class>ReferenceDelayResolver</class>
<defaultDelay>3000</defaultDelay>
<delayReferencePatterns>
  <delayReferencePattern>
    <delay>10000</delay>
    <pattern>.*.pdf</pattern>
  </delayReferencePattern>
</delayReferencePatterns>

Usage

Full configuration skeleton, with every option and its default

YAML

class: ReferenceDelayResolver
defaultDelay: 0
delayReferencePatterns:
  - delay: 0
    pattern: string
ignoreRobotsCrawlDelay: false
scope: CRAWLER

JSON

{
  "class": "ReferenceDelayResolver",
  "defaultDelay": 0,
  "delayReferencePatterns": [
    {
      "delay": 0,
      "pattern": "string"
    }
  ],
  "ignoreRobotsCrawlDelay": false,
  "scope": "CRAWLER"
}

XML

<delayResolver>
  <class>ReferenceDelayResolver</class>
  <defaultDelay>0</defaultDelay>
  <delayReferencePatterns>
    <delayReferencePattern>
      <delay>0</delay>
      <pattern>string</pattern>
    </delayReferencePattern>
  </delayReferencePatterns>
  <ignoreRobotsCrawlDelay>false</ignoreRobotsCrawlDelay>
  <scope>CRAWLER</scope>
</delayResolver>

Properties

Property	Type	Required	Default
defaultDelay	integer	No	3000
delayReferencePatterns	DelayReferencePattern[]	No	-
ignoreRobotsCrawlDelay	boolean	No	false
scope	enum	No	CRAWLER

Property Details

defaultDelay

The default delay in milliseconds.

delayReferencePatterns

Use different delays by corresponding URLs.

ignoreRobotsCrawlDelay

By default the crawl delay specified in robots.txt takes precedence over custom settings. Check to disable robot delay instructions.

scope

The delay scope.

Allowed Values

• CRAWLER
• SITE
• THREAD