GenericDelayResolver

Apply delay per time schedule.

crawler/webJavadoc: GenericDelayResolver

Notes

Applies configurable crawl delays based on time schedules. Supports a default delay, optional time-range-based schedules (by day of week, day of month, or time of day), and an option to ignore the crawl delay specified in robots.txt.

Examples

The following example set the minimum delay between each document download on a given site to 5 seconds, no matter what the crawler robots.txt may say, excepted on weekend, where it is more agressive (1 second).

YAML

class: GenericDelayResolver
defaultDelay: 5000
ignoreRobotsCrawlDelay: true
scope: SITE
schedules:
  - dayOfWeekRange:
      start: SAT
      end: SUN
    delay: 1000

JSON

{
  "class": "GenericDelayResolver",
  "defaultDelay": 5000,
  "ignoreRobotsCrawlDelay": true,
  "scope": "SITE",
  "schedules": [
    {
      "dayOfWeekRange": {
        "start": "SAT",
        "end": "SUN"
      },
      "delay": 1000
    }
  ]
}

XML

<class>GenericDelayResolver</class>
<defaultDelay>5000</defaultDelay>
<ignoreRobotsCrawlDelay>true</ignoreRobotsCrawlDelay>
<scope>SITE</scope>
<schedules>
  <dayOfWeekRange>
    <start>SAT</start>
    <end>SUN</end>
  </dayOfWeekRange>
  <delay>1000</delay>
</schedules>

Usage

Full configuration skeleton, with every option and its default

YAML

class: GenericDelayResolver
defaultDelay: 0
ignoreRobotsCrawlDelay: false
schedules:
  - dayOfMonthRange: {}
    dayOfWeekRange: {}
    delay: 0
    timeRange: {}
scope: CRAWLER

JSON

{
  "class": "GenericDelayResolver",
  "defaultDelay": 0,
  "ignoreRobotsCrawlDelay": false,
  "schedules": [
    {
      "dayOfMonthRange": {},
      "dayOfWeekRange": {},
      "delay": 0,
      "timeRange": {}
    }
  ],
  "scope": "CRAWLER"
}

XML

<delayResolver>
  <class>GenericDelayResolver</class>
  <defaultDelay>0</defaultDelay>
  <ignoreRobotsCrawlDelay>false</ignoreRobotsCrawlDelay>
  <schedules>
    <schedule>
      <dayOfMonthRange/>
      <dayOfWeekRange/>
      <delay>0</delay>
      <timeRange/>
    </schedule>
  </schedules>
  <scope>CRAWLER</scope>
</delayResolver>

Properties

Property	Type	Required	Default
defaultDelay	integer	No	3000
ignoreRobotsCrawlDelay	boolean	No	false
schedules	DelaySchedule[]	No	-
scope	enum	No	CRAWLER

Property Details

defaultDelay

The default delay in milliseconds.

ignoreRobotsCrawlDelay

By default the crawl delay specified in robots.txt takes precedence over custom settings. Check to disable robot delay instructions.

schedules

Have different delays at different time.

scope

The delay scope.

Allowed Values

• CRAWLER
• SITE
• THREAD