Package com.norconex.crawler.web.doc.operations.robot.impl

package com.norconex.crawler.web.doc.operations.robot.impl

Related Packages

Package	Description
com.norconex.crawler.web.doc.operations.robot

Classes

Class	Description
StandardRobotsMetaProvider	Implementation of RobotsMetaProvider as per X-Robots-Tag and ROBOTS standards.
StandardRobotsMetaProviderConfig	Configuration for StandardRobotsMetaProvider.
StandardRobotsTxtProvider	Implementation of RobotsTxtProvider as per the robots.txt standard described at http://www.robotstxt.org/robotstxt.html.