GenericSitemapResolver

Parses sitemap XML files to extract URLs for crawling.

crawler/webJavadoc: GenericSitemapResolver

Notes

Parses and processes sitemap XML files to extract URLs for crawling. Supports lenient parsing mode for sitemaps that do not strictly conform to the sitemap protocol.

Usage

YAML

class: GenericSitemapResolver
lenient: false

JSON

{
  "class": "GenericSitemapResolver",
  "lenient": false
}

XML

<sitemapResolver>
  <class>GenericSitemapResolver</class>
  <lenient>false</lenient>
</sitemapResolver>

Properties

Property	Type	Required	Default
lenient	boolean	No	false

Property Details

lenient

When enabled, the sitemap parser is more tolerant of sitemaps that do not strictly conform to the sitemap protocol specification.