Package com.norconex.crawler.web.doc.operations.link.impl

Class HtmlLinkExtractor

java.lang.Object
com.norconex.crawler.web.doc.operations.link.impl.HtmlLinkExtractor
All Implemented Interfaces:
Configurable<HtmlLinkExtractorConfig>, LinkExtractor
public class HtmlLinkExtractor extends Object implements LinkExtractor, Configurable<HtmlLinkExtractorConfig>

A memory efficient HTML link extractor.

This link extractor uses regular expressions to extract links. It does so on a chunk of text at a time, so that large files are not fully loaded into memory. If you prefer a more flexible implementation that loads the DOM model in memory to perform link extraction, consider using DomLinkExtractor.

Applicable documents

By default, this extractor will only be applied on documents matching one of the content-types specified by CommonMatchers.HTML_CONTENT_TYPES

You can specify your own content types or other restrictions with HtmlLinkExtractorConfig.setContentTypeMatcher(com.norconex.commons.lang.text.TextMatcher). Make sure they represent a file with HTML-like markup tags containing URLs. For documents that are just too different, consider implementing your own LinkExtractor instead. Removing the default values and define no content types will have for effect to try to extract URLs from all files (usually a bad idea).

Tags attributes

URLs are assumed to be contained within valid tags or tag attributes. The default tags and attributes used are (tag.attribute): 
 a.href, frame.src, iframe.src, img.src, meta.http-equiv
You can specify your own set of tags and attributes to have different ones used for extracting URLs. For an elaborated set, you can combine the above with your own list or use any of the following suggestions (tag.attribute): 
 applet.archive,   applet.codebase,  area.href,         audio.src,
 base.href,        blockquote.cite,  body.background,   button.formaction,
 command.icon,     del.cite,         embed.src,         form.action,
 frame.longdesc,   head.profile,     html.manifest,     iframe.longdesc,
 img.longdesc,     img.usemap,       input.formaction,  input.src,
 input.usemap,     ins.cite,         link.href,         object.archive,
 object.classid,   object.codebase,  object.data,       object.usemap,
 q.cite,           script.src,       source.src,        video.poster,
 video.src

The meta.http-equiv is treated differently. Only if the "http-equiv" value is "refresh" and a "content" attribute with a URL exist that it will be extracted. The "object" and "applet" tags can have multiple URLs.

It is possible to identify a tag only as the holder of a URL (without attributes). The tag body value will be used as the URL.

Referrer data

Some "referrer" information is derived from the each link and stored as metadata in the document they point to. These may vary for each link, but they are normally prefixed with WebDocMetadata.REFERRER_LINK_PREFIX.

The referrer data is always stored.

Character encoding

This extractor will by default attempt to detect the encoding of the a page when extracting links and referrer information. If no charset could be detected, it falls back to UTF-8. It is also possible to dictate which encoding to use with HtmlLinkExtractorConfig.setCharset(java.nio.charset.Charset).

"nofollow"

By default, a regular HTML link having the "rel" attribute set to "nofollow" won't be extracted (e.g. <a href="x.html" rel="nofollow" ...>). To force its extraction (and ensure it is followed) you can set HtmlLinkExtractorConfig.setIgnoreNofollow(boolean) to true.

URL Fragments

While extractor preserves hashtag characters (#) found in URLs and every characters after it, the default URL normalizer (GenericUrlNormalizer) will strip it by default.

The URL specification says hashtags are used to represent fragments only. That is, to quickly jump to a specific section of the page the URL represents. Under normal circumstances, keeping the URL fragments usually leads to duplicates documents being fetched (same URL but different fragment) and they should be stripped. Unfortunately, there are sites not following the URL standard and using hashtags as a regular part of a URL (i.e. different hashtags point to different web pages). It may be essential when crawling these sites to keep the URL fragments. This can be done by making sure the URL normalizer does not strip them.

By default, contextual information is kept about the HTML/XML mark-up tag from which a link is extracted (e.g., tag name and attributes). That information gets stored as metadata in the target document. If you want to limit the quantity of information extracted/stored, you can disable this feature by setting HtmlLinkExtractorConfig.setIgnoreLinkData(boolean) to true.

URL Schemes

Only valid schemes are extracted for absolute URLs. By default, those are http, https, and ftp. You can specify your own list of supported protocols with HtmlLinkExtractorConfig.setSchemes(List).

HTML/XML Comments

URLs found in <!-- comments --> are not extracted by default. To enable URL extraction from comments, use HtmlLinkExtractorConfig.setCommentsEnabled(boolean)

You can identify portions of a document where links should be extracted or ignored with HtmlLinkExtractorConfig.setExtractBetweens(List) and HtmlLinkExtractorConfig.setNoExtractBetweens(List). Eligible content for link extraction is identified first, and content to exclude is done on that subset.

You can further limit link extraction to specific area by using selector-syntax to do so, with HtmlLinkExtractorConfig.setExtractSelectors(List) and HtmlLinkExtractorConfig.setNoExtractSelectors(List).