Package com.norconex.crawler.web.doc.operations.robot

Class RobotsTxt

java.lang.Object

com.norconex.crawler.web.doc.operations.robot.RobotsTxt

public class RobotsTxt
extends Object

Immutable holder of robots.txt rules.

Nested Class Summary

Nested Classes

Modifier and Type	Class	Description
static class	RobotsTxt.RobotsTxtBuilder

Field Summary

Fields

Modifier and Type	Field	Description
static final float	UNSPECIFIED_CRAWL_DELAY

Method Summary

Modifier and Type	Method	Description
static RobotsTxt.RobotsTxtBuilder	builder()
protected boolean	canEqual(Object other)
boolean	equals(Object o)
List<RobotsTxtFilter>	getAllowFilters()	Gets "Allow" filters.
float	getCrawlDelay()
List<RobotsTxtFilter>	getDisallowFilters()	Gets "Disallow" filters.
List<String>	getSitemapLocations()
int	hashCode()
boolean	isEmpty()
String	toString()

Methods inherited from class java.lang.Object

clone, finalize, getClass, notify, notifyAll, wait, wait, wait

Field Details

UNSPECIFIED_CRAWL_DELAY

public static final float UNSPECIFIED_CRAWL_DELAY

See Also:

• Constant Field Values

Method Details

getDisallowFilters

public List<RobotsTxtFilter> getDisallowFilters()

Gets "Disallow" filters.

Returns:

disallow filters (never null)

Since:

2.4.0

getAllowFilters

public List<RobotsTxtFilter> getAllowFilters()

Gets "Allow" filters.

Returns:

allow filters (never null)

Since:

2.4.0

getSitemapLocations

public List<String> getSitemapLocations()

getCrawlDelay

public float getCrawlDelay()

isEmpty

public boolean isEmpty()

builder

public static RobotsTxt.RobotsTxtBuilder builder()

toString

public String toString()

Overrides:

toString in class Object

equals

public boolean equals(Object o)

Overrides:

equals in class Object

canEqual

protected boolean canEqual(Object other)

hashCode

public int hashCode()

Overrides:

hashCode in class Object