Package com.norconex.crawler.web.doc.operations.robot.impl

Class StandardRobotsMetaProvider

java.lang.Object

com.norconex.crawler.web.doc.operations.robot.impl.StandardRobotsMetaProvider

All Implemented Interfaces:

Configurable<StandardRobotsMetaProviderConfig>, RobotsMetaProvider

public class StandardRobotsMetaProvider
extends Object
implements RobotsMetaProvider, Configurable<StandardRobotsMetaProviderConfig>

Implementation of RobotsMetaProvider as per X-Robots-Tag and ROBOTS standards. Extracts robots information from "ROBOTS" meta tag in an HTML page or "X-Robots-Tag" tag in the HTTP header (see https://developers.google.com/webmasters/control-crawl-index/docs/robots_meta_tag and http://www.robotstxt.org/meta.html).

If you specified a prefix for the HTTP headers, make sure to specify it again here or the robots meta tags will not be found.

If robots instructions are provided in both the HTML page and HTTP header, the ones in HTML page will take precedence, and the ones in HTTP header will be ignored.

Field Summary

Fields inherited from interface com.norconex.commons.lang.config.Configurable

PROPERTY

Constructor Summary

Constructors

Constructor	Description
StandardRobotsMetaProvider()

Method Summary

Modifier and Type	Method	Description
protected boolean	canEqual(Object other)
boolean	equals(Object o)
StandardRobotsMetaProviderConfig	getConfiguration()
RobotsMeta	getRobotsMeta(Reader document, String documentUrl, ContentType contentType, Properties httpHeaders)	Extracts Robots meta information for a page, if any.
int	hashCode()
String	toString()

Methods inherited from class java.lang.Object

clone, finalize, getClass, notify, notifyAll, wait, wait, wait

Constructor Details

StandardRobotsMetaProvider

public StandardRobotsMetaProvider()

Method Details

getRobotsMeta

public RobotsMeta getRobotsMeta(Reader document,
 String documentUrl,
 ContentType contentType,
 Properties httpHeaders)
                         throws IOException

Description copied from interface: RobotsMetaProvider

Extracts Robots meta information for a page, if any.

Specified by:

getRobotsMeta in interface RobotsMetaProvider

Parameters:

document - the document

documentUrl - document url

contentType - the document content type

httpHeaders - the document HTTP Headers

Returns:

robots meta instance

Throws:

IOException - problem reading the document

equals

public boolean equals(Object o)

Overrides:

equals in class Object

canEqual

protected boolean canEqual(Object other)

hashCode

public int hashCode()

Overrides:

hashCode in class Object

toString

public String toString()

Overrides:

toString in class Object

getConfiguration

public StandardRobotsMetaProviderConfig getConfiguration()

Specified by:

getConfiguration in interface Configurable<StandardRobotsMetaProviderConfig>