Uses of Class
com.norconex.crawler.web.doc.operations.robot.RobotsTxt

Packages that use RobotsTxt

Package	Description
com.norconex.crawler.web.doc.operations.delay
com.norconex.crawler.web.doc.operations.delay.impl
com.norconex.crawler.web.doc.operations.robot
com.norconex.crawler.web.doc.operations.robot.impl
com.norconex.crawler.web.util

Uses of RobotsTxt in com.norconex.crawler.web.doc.operations.delay

Methods in com.norconex.crawler.web.doc.operations.delay with parameters of type RobotsTxt

Modifier and Type	Method	Description
void	DelayResolver.delay(RobotsTxt robotsTxt, String url)	Delay crawling activities (if applicable).

Uses of RobotsTxt in com.norconex.crawler.web.doc.operations.delay.impl

Methods in com.norconex.crawler.web.doc.operations.delay.impl with parameters of type RobotsTxt

Modifier and Type	Method	Description
void	AbstractDelayResolver.delay(RobotsTxt robotsTxt, String url)

Uses of RobotsTxt in com.norconex.crawler.web.doc.operations.robot

Methods in com.norconex.crawler.web.doc.operations.robot that return RobotsTxt

Modifier and Type	Method	Description
RobotsTxt	RobotsTxt.RobotsTxtBuilder.build()
RobotsTxt	RobotsTxtProvider.getRobotsTxt(Fetcher fetchClient, String url)	Gets robots.txt rules.

Uses of RobotsTxt in com.norconex.crawler.web.doc.operations.robot.impl

Methods in com.norconex.crawler.web.doc.operations.robot.impl that return RobotsTxt

Modifier and Type	Method	Description
RobotsTxt	StandardRobotsTxtProvider.getRobotsTxt(Fetcher fetcher, String url)
protected RobotsTxt	StandardRobotsTxtProvider.parseRobotsTxt(InputStream is, String url, String userAgent)

Uses of RobotsTxt in com.norconex.crawler.web.util

Methods in com.norconex.crawler.web.util that return RobotsTxt

Modifier and Type	Method	Description
static RobotsTxt	Web.robotsTxt(CrawlerContext crawler, String reference)