StandardRobotsTxtProvider

Fetches and enforces robots.txt crawl rules from web sites.

crawler/webJavadoc: StandardRobotsTxtProvider

Notes

Fetches and parses robots.txt files from web sites according to the Robots Exclusion Standard. Crawl directives found in robots.txt are applied to restrict or allow access to specific URL paths.

Usage

YAML

class: StandardRobotsTxtProvider
noop: string

JSON

{
  "class": "StandardRobotsTxtProvider",
  "noop": "string"
}

XML

<robotsTxtProvider>
  <class>StandardRobotsTxtProvider</class>
  <noop>string</noop>
</robotsTxtProvider>