RobotsTxtProvider
Supported implementations for Robots Txt Provider.
Notes
Reads and enforces crawl rules from robots.txt files found on web sites. Select one of the supported implementations to configure how robots.txt directives are respected. Set to null to disable robots.txt support entirely.
Examples
The following example disables support for "robots.txt" files present on web sites.
- YAML
- JSON
- XML
robotsTxtProvider: null
{
"robotsTxtProvider": null
}
<robotsTxtProvider/>
Implementations
- StandardRobotsTxtProvider Default