Package com.norconex.crawler.web.doc.operations.robot

package com.norconex.crawler.web.doc.operations.robot

Related Packages

Package	Description
com.norconex.crawler.web.doc.operations.robot.impl

Class	Description
RobotsMeta
RobotsMetaProvider	Responsible for extracting robot information from a page.
RobotsTxt	Immutable holder of robots.txt rules.
RobotsTxt.RobotsTxtBuilder
RobotsTxtFilter	Holds a robots.txt rule.
RobotsTxtProvider	Given a URL, extract any "robots.txt" rules.