StandardRobotsMetaProvider

Reads robots directives from HTML meta tags and HTTP response headers.

crawler/webJavadoc: StandardRobotsMetaProvider

Notes

Reads robots directives from HTML <meta name="robots"> tags and HTTP response headers (such as X-Robots-Tag). Supports a configurable prefix for captured response header fields.

Usage

YAML

class: StandardRobotsMetaProvider
headersPrefix: string

JSON

{
  "class": "StandardRobotsMetaProvider",
  "headersPrefix": "string"
}

XML

<robotsMetaProvider>
  <class>StandardRobotsMetaProvider</class>
  <headersPrefix>string</headersPrefix>
</robotsMetaProvider>

Properties

Property	Type	Required	Default
headersPrefix	string	No	-

Property Details

headersPrefix

An optional prefix to prepend to HTTP response header field names when storing robots metadata as document metadata. Default is null (no prefix).