GenericSitemapLocator

Locates sitemap files for crawled websites.

crawler/webJavadoc: GenericSitemapLocator

Notes

Locates sitemap files for crawled websites by looking at known sitemap paths and optionally reading the sitemap location from robots.txt. Supports configuring additional custom sitemap paths.

Usage

YAML

class: GenericSitemapLocator
paths:
  - string
robotsTxtSitemapDisabled: false

JSON

{
  "class": "GenericSitemapLocator",
  "paths": [
    "string"
  ],
  "robotsTxtSitemapDisabled": false
}

XML

<sitemapLocator>
  <class>GenericSitemapLocator</class>
  <paths>
    <path>string</path>
  </paths>
  <robotsTxtSitemapDisabled>false</robotsTxtSitemapDisabled>
</sitemapLocator>

Properties

Property	Type	Required	Default
paths	string[]	No	[/sitemap.xml, /sitemap_index.xml]
robotsTxtSitemapDisabled	boolean	No	false

Property Details

paths

Domain-relative URL paths where to look for sitemaps when not supplied as a start reference or part of a web site robots.txt file. Defaults to /sitemap.xml and /sitemap_index.xml.

robotsTxtSitemapDisabled

Whether to disable checking for sitemap locations in a web site robots.txt file. When disabled, only the configured paths (or their defaults) are used to locate sitemaps.