Package com.norconex.crawler.web.doc.operations.robot.impl

Class StandardRobotsTxtProvider

java.lang.Object

com.norconex.crawler.core.event.listeners.CrawlerLifeCycleListener

com.norconex.crawler.web.doc.operations.robot.impl.StandardRobotsTxtProvider

All Implemented Interfaces:

EventListener<Event>, RobotsTxtProvider, EventListener, Consumer<Event>

public class StandardRobotsTxtProvider
extends CrawlerLifeCycleListener
implements RobotsTxtProvider

Implementation of RobotsTxtProvider as per the robots.txt standard described at http://www.robotstxt.org/robotstxt.html.

Constructor Summary

Constructors

Constructor	Description
StandardRobotsTxtProvider()

Method Summary

Modifier and Type	Method	Description
protected boolean	canEqual(Object other)
boolean	equals(Object o)
RobotsTxt	getRobotsTxt(Fetcher fetcher, String url)	Gets robots.txt rules.
int	hashCode()
protected void	onCrawlerCrawlBegin(CrawlerEvent event)
protected RobotsTxt	parseRobotsTxt(InputStream is, String url, String userAgent)
String	toString()

Methods inherited from class com.norconex.crawler.core.event.listeners.CrawlerLifeCycleListener

accept, onCrawlerCleanBegin, onCrawlerCleanEnd, onCrawlerCrawlEnd, onCrawlerError, onCrawlerEvent, onCrawlerStopBegin, onCrawlerStopEnd

Methods inherited from class java.lang.Object

clone, finalize, getClass, notify, notifyAll, wait, wait, wait

Methods inherited from interface java.util.function.Consumer

andThen

Constructor Details

StandardRobotsTxtProvider

public StandardRobotsTxtProvider()

Method Details

onCrawlerCrawlBegin

protected void onCrawlerCrawlBegin(CrawlerEvent event)

Overrides:

onCrawlerCrawlBegin in class CrawlerLifeCycleListener

getRobotsTxt

public RobotsTxt getRobotsTxt(Fetcher fetcher,
 String url)

Description copied from interface: RobotsTxtProvider

Gets robots.txt rules. This method signature changed in 1.3 to include the userAgent.

Specified by:

getRobotsTxt in interface RobotsTxtProvider

Parameters:

fetcher - http fetcher executor to grab robots.txt

url - the URL to derive the robots.txt from

Returns:

robots.txt rules

parseRobotsTxt

protected RobotsTxt parseRobotsTxt(InputStream is,
 String url,
 String userAgent)
                            throws IOException

Throws:

IOException

equals

public boolean equals(Object o)

Overrides:

equals in class CrawlerLifeCycleListener

canEqual

protected boolean canEqual(Object other)

Overrides:

canEqual in class CrawlerLifeCycleListener

hashCode

public int hashCode()

Overrides:

hashCode in class CrawlerLifeCycleListener

toString

public String toString()

Overrides:

toString in class CrawlerLifeCycleListener