DelayResolverScope

Controls the scope at which crawl delays are applied.

crawler/webJavadoc: DelayResolverScope

Notes

Specifies whether a crawl delay is enforced per crawler instance, per remote site (host), or per crawling thread. Choosing the appropriate scope ensures polite crawling behavior without unnecessarily slowing the overall crawl.

Usage

YAML

DelayResolverScope: CRAWLER

JSON

{
  "DelayResolverScope": "CRAWLER"
}

XML

<delayResolverScope/>

Allowed Values

• CRAWLER
• SITE
• THREAD