Package com.norconex.crawler.web.doc.operations.link.impl

Class RegexLinkExtractor

java.lang.Object

com.norconex.crawler.web.doc.operations.link.impl.RegexLinkExtractor

All Implemented Interfaces:

Configurable<RegexLinkExtractorConfig>, LinkExtractor

public class RegexLinkExtractor
extends Object
implements LinkExtractor, Configurable<RegexLinkExtractorConfig>

Link extractor using regular expressions to extract links found in text documents. Relative links are resolved to the document URL. For HTML documents, it is best advised to use the HtmlLinkExtractor or DomLinkExtractor, which addresses many cases specific to HTML.

Applicable documents

By default, this extractor will extract URLs only in documents having their content type matching this regular expression:

 text/.*
 

You can specify your own restrictions using RegexLinkExtractorConfig.getRestrictions(), but make sure they represent text files.

Referrer data

The following referrer information is stored as metadata in each document represented by the extracted URLs:

• Referrer reference: The reference (URL) of the page where the link to a document was found. Metadata value is WebDocMetadata.REFERRER_REFERENCE.

Character encoding

This extractor will by default attempt to detect the encoding of the a page when extracting links and referrer information. If no charset could be detected, it falls back to UTF-8. It is also possible to dictate which encoding to use with RegexLinkExtractorConfig.setCharset(java.nio.charset.Charset).

Since:

2.7.0

Field Summary

Fields inherited from interface com.norconex.commons.lang.config.Configurable

PROPERTY

Constructor Summary

Constructors

Constructor	Description
RegexLinkExtractor()

Method Summary

Modifier and Type	Method	Description
protected boolean	canEqual(Object other)
boolean	equals(Object o)
Set<Link>	extractLinks(Doc doc)
RegexLinkExtractorConfig	getConfiguration()
int	hashCode()
String	toString()

Methods inherited from class java.lang.Object

clone, finalize, getClass, notify, notifyAll, wait, wait, wait

Constructor Details

RegexLinkExtractor

public RegexLinkExtractor()

Method Details

extractLinks

public Set<Link> extractLinks(Doc doc)
                       throws IOException

Specified by:

extractLinks in interface LinkExtractor

Throws:

IOException

equals

public boolean equals(Object o)

Overrides:

equals in class Object

canEqual

protected boolean canEqual(Object other)

hashCode

public int hashCode()

Overrides:

hashCode in class Object

toString

public String toString()

Overrides:

toString in class Object

getConfiguration

public RegexLinkExtractorConfig getConfiguration()

Specified by:

getConfiguration in interface Configurable<RegexLinkExtractorConfig>