Uses of Interface
com.norconex.crawler.web.doc.operations.robot.RobotsTxtProvider

Packages that use RobotsTxtProvider

Package	Description
com.norconex.crawler.web
com.norconex.crawler.web.doc.operations.robot.impl

Uses of RobotsTxtProvider in com.norconex.crawler.web

Methods in com.norconex.crawler.web that return RobotsTxtProvider

Modifier and Type	Method	Description
RobotsTxtProvider	WebCrawlerConfig.getRobotsTxtProvider()	The provider of robots.txt rules for a site (if applicable).

Methods in com.norconex.crawler.web with parameters of type RobotsTxtProvider

Modifier and Type	Method	Description
WebCrawlerConfig	WebCrawlerConfig.setRobotsTxtProvider(RobotsTxtProvider robotsTxtProvider)	The provider of robots.txt rules for a site (if applicable).

Uses of RobotsTxtProvider in com.norconex.crawler.web.doc.operations.robot.impl

Classes in com.norconex.crawler.web.doc.operations.robot.impl that implement RobotsTxtProvider

Modifier and Type	Class	Description
class	StandardRobotsTxtProvider	Implementation of RobotsTxtProvider as per the robots.txt standard described at http://www.robotstxt.org/robotstxt.html.