Package com.norconex.crawler.web.doc.operations.link.impl

Class DomLinkExtractor

java.lang.Object
com.norconex.crawler.web.doc.operations.link.impl.DomLinkExtractor
All Implemented Interfaces:
Configurable<DomLinkExtractorConfig>, LinkExtractor
public class DomLinkExtractor extends Object implements LinkExtractor, Configurable<DomLinkExtractorConfig>

Extracts links from a Document Object Model (DOM) representation of an HTML, XHTML, or XML document content based on values of matching elements and attributes.

In order to construct a DOM tree, text is loaded entirely into memory. It uses the document content by default, but it can also come from specified metadata fields. Use this filter with caution if you know you'll need to parse huge files. Use the HtmlLinkExtractor instead if this is a concern.

The jsoup parser library is used to load a document content into a DOM tree. Elements are referenced using a CSS or JQuery-like syntax.

This link extractor is normally used before importing.

When used before importing this class attempts to detect the content character encoding unless the character encoding was specified using DomLinkExtractorConfig.setCharset(java.nio.charset.Charset). Since document parsing converts content to UTF-8, UTF-8 is always assumed when used as a post-parse handler.

You can specify which DOM parser to use when reading documents. The default is "html" and will normalize the content as HTML. This is generally a desired behavior, but this can sometimes have your selector fail. If you encounter this problem, try switching to "xml" parser, which does not attempt normalization on the content. The drawback with "xml" is you may not get all HTML-specific selector options to work. If you know you are dealing with XML to begin with, specifying "xml" should be a good option.

You can define as many JSoup "selectors" as desired. All values matched by a selector will be extracted as a URL.

It is possible to control what gets extracted exactly for matching purposes thanks to the "extract" argument expected with every selector. See DomUtil for possible values.

When not specified, the default is "text".

The default selectors / extract strategies are:

  • a[href] / attr(href)
  • [src] / attr(src)
  • link[href] / attr(href)
  • meta[http-equiv='refresh'] / attr(content)

For any extracted link values, this extractor will perform minimal heuristics to clean extra content not part of a regular URL. For instance, it will only keep what is after url= when dealing with <meta http-equiv refresh URLs. It will also trim white spaces.

By default, contextual information is kept about the HTML/XML mark-up tag from which a link is extracted (e.g., tag name and attributes). That information gets stored as metadata in the target document. If you want to limit the quantity of information extracted/stored, you can disable this feature by setting DomLinkExtractorConfig.setIgnoreLinkData(boolean) to true.

URL Schemes

Only valid schemes are extracted for absolute URLs. By default, those are http, https, and ftp. You can specify your own list of supported protocols with DomLinkExtractorConfig.setSchemes(java.util.List).

Applicable documents

By default, this extractor will only be applied on documents matching one of these content types: CommonMatchers.DOM_CONTENT_TYPES.

"nofollow"

By default, a regular HTML link having the "rel" attribute set to "nofollow" won't be extracted (e.g. <a href="x.html" rel="nofollow" ...>). To force its extraction (and ensure it is followed) you can set DomLinkExtractorConfig.setIgnoreNofollow(boolean) to true.

Since:
3.0.0