StdRobotsTxtFilter

A reference filter that enforces robots.txt rules on crawled URLs.

crawler/webJavadoc: StdRobotsTxtFilter

Notes

A reference filter that applies robots.txt rules to crawled URLs. URLs that are disallowed by the site's robots.txt directives are excluded from crawling.

Usage

YAML

onMatch: INCLUDE
valueMatcher:
  ignoreCase: false
  ignoreDiacritic: false
  matchEmpty: false
  method: BASIC
  negateMatches: false
  partial: false
  pattern: string
  replaceAll: false
  trim: false

JSON

{
  "onMatch": "INCLUDE",
  "valueMatcher": {
    "ignoreCase": false,
    "ignoreDiacritic": false,
    "matchEmpty": false,
    "method": "BASIC",
    "negateMatches": false,
    "partial": false,
    "pattern": "string",
    "replaceAll": false,
    "trim": false
  }
}

XML

<stdRobotsTxtFilter>
  <onMatch>INCLUDE</onMatch>
  <valueMatcher>
    <ignoreCase>false</ignoreCase>
    <ignoreDiacritic>false</ignoreDiacritic>
    <matchEmpty>false</matchEmpty>
    <method>BASIC</method>
    <negateMatches>false</negateMatches>
    <partial>false</partial>
    <pattern>string</pattern>
    <replaceAll>false</replaceAll>
    <trim>false</trim>
  </valueMatcher>
</stdRobotsTxtFilter>

Properties

Property	Type	Required	Default
onMatch	enum	No	-
valueMatcher	TextMatcher	No	TextMatcher

Property Details

onMatch

The action to take when a URL matches the robots.txt rule. INCLUDE means the URL is allowed; EXCLUDE means the URL is disallowed and will be skipped.

Allowed Values

• INCLUDE
• EXCLUDE

valueMatcher

Matches document field values based on specified patterns and methods. Use this to restrict the handler to operate only on fields whose values satisfy the matcher criteria.