HtmlLinkExtractor

Extracts hyperlinks from HTML documents by scanning tag attributes.

crawler/webJavadoc: HtmlLinkExtractor

Notes

Extracts links from HTML documents by scanning known HTML tags and attributes for URLs. Supports custom tag/attribute combinations, CSS selector-based inclusion and exclusion zones, content-type restrictions, nofollow handling, and URL length limits.

Examples

The following example adds URLs from <script> tags to the default list of URLs to be extracted.

YAML

class: HtmlLinkExtractor
tagAttribs:
  a:
    - href
  frame:
    - src
  iframe:
    - src
  img:
    - src
  meta:
    - http-equiv
  script:
    - src

JSON

{
  "class": "HtmlLinkExtractor",
  "tagAttribs": {
    "a": [
      "href"
    ],
    "frame": [
      "src"
    ],
    "iframe": [
      "src"
    ],
    "img": [
      "src"
    ],
    "meta": [
      "http-equiv"
    ],
    "script": [
      "src"
    ]
  }
}

XML

<class>HtmlLinkExtractor</class>
<tagAttribs>
  <a>href</a>
  <frame>src</frame>
  <iframe>src</iframe>
  <img>src</img>
  <meta>http-equiv</meta>
  <script>src</script>
</tagAttribs>

Usage

Full configuration skeleton, with every option and its default

YAML

class: HtmlLinkExtractor
charset: string
commentsEnabled: false
contentTypeMatcher:
  ignoreCase: false
  ignoreDiacritic: false
  matchEmpty: false
  method: BASIC
  negateMatches: false
  partial: false
  pattern: string
  replaceAll: false
  trim: false
extractBetweens:
  - end: string
    ignoreCase: false
    start: string
extractSelectors:
  - string
fieldMatcher:
  ignoreCase: false
  ignoreDiacritic: false
  matchEmpty: false
  method: BASIC
  negateMatches: false
  partial: false
  pattern: string
  replaceAll: false
  trim: false
ignoreLinkData: false
ignoreNofollow: false
maxURLLength: 0
noExtractBetweens:
  - end: string
    ignoreCase: false
    start: string
noExtractSelectors:
  - string
restrictions:
  - fieldMatcher: {}
    valueMatcher: {}
schemes:
  - string
tagAttribs:
  caseInsensitiveKeys: false

JSON

{
  "class": "HtmlLinkExtractor",
  "charset": "string",
  "commentsEnabled": false,
  "contentTypeMatcher": {
    "ignoreCase": false,
    "ignoreDiacritic": false,
    "matchEmpty": false,
    "method": "BASIC",
    "negateMatches": false,
    "partial": false,
    "pattern": "string",
    "replaceAll": false,
    "trim": false
  },
  "extractBetweens": [
    {
      "end": "string",
      "ignoreCase": false,
      "start": "string"
    }
  ],
  "extractSelectors": [
    "string"
  ],
  "fieldMatcher": {
    "ignoreCase": false,
    "ignoreDiacritic": false,
    "matchEmpty": false,
    "method": "BASIC",
    "negateMatches": false,
    "partial": false,
    "pattern": "string",
    "replaceAll": false,
    "trim": false
  },
  "ignoreLinkData": false,
  "ignoreNofollow": false,
  "maxURLLength": 0,
  "noExtractBetweens": [
    {
      "end": "string",
      "ignoreCase": false,
      "start": "string"
    }
  ],
  "noExtractSelectors": [
    "string"
  ],
  "restrictions": [
    {
      "fieldMatcher": {},
      "valueMatcher": {}
    }
  ],
  "schemes": [
    "string"
  ],
  "tagAttribs": {
    "caseInsensitiveKeys": false
  }
}

XML

<htmlLinkExtractor>
  <class>HtmlLinkExtractor</class>
  <charset>string</charset>
  <commentsEnabled>false</commentsEnabled>
  <contentTypeMatcher>
    <ignoreCase>false</ignoreCase>
    <ignoreDiacritic>false</ignoreDiacritic>
    <matchEmpty>false</matchEmpty>
    <method>BASIC</method>
    <negateMatches>false</negateMatches>
    <partial>false</partial>
    <pattern>string</pattern>
    <replaceAll>false</replaceAll>
    <trim>false</trim>
  </contentTypeMatcher>
  <extractBetweens>
    <extractBetween>
      <end>string</end>
      <ignoreCase>false</ignoreCase>
      <start>string</start>
    </extractBetween>
  </extractBetweens>
  <extractSelectors>
    <extractSelector>string</extractSelector>
  </extractSelectors>
  <fieldMatcher>
    <ignoreCase>false</ignoreCase>
    <ignoreDiacritic>false</ignoreDiacritic>
    <matchEmpty>false</matchEmpty>
    <method>BASIC</method>
    <negateMatches>false</negateMatches>
    <partial>false</partial>
    <pattern>string</pattern>
    <replaceAll>false</replaceAll>
    <trim>false</trim>
  </fieldMatcher>
  <ignoreLinkData>false</ignoreLinkData>
  <ignoreNofollow>false</ignoreNofollow>
  <maxURLLength>0</maxURLLength>
  <noExtractBetweens>
    <noExtractBetween>
      <end>string</end>
      <ignoreCase>false</ignoreCase>
      <start>string</start>
    </noExtractBetween>
  </noExtractBetweens>
  <noExtractSelectors>
    <noExtractSelector>string</noExtractSelector>
  </noExtractSelectors>
  <restrictions>
    <restriction>
      <fieldMatcher/>
      <valueMatcher/>
    </restriction>
  </restrictions>
  <schemes>
    <scheme>string</scheme>
  </schemes>
  <tagAttribs>
    <caseInsensitiveKeys>false</caseInsensitiveKeys>
  </tagAttribs>
</htmlLinkExtractor>

Properties

Property	Type	Required	Default
charset	string	No	-
commentsEnabled	boolean	No	false
contentTypeMatcher	TextMatcher	No	TextMatcher
extractBetweens	RegexPair[]	No	-
extractSelectors	string[]	No	-
fieldMatcher	TextMatcher	No	TextMatcher
ignoreLinkData	boolean	No	false
ignoreNofollow	boolean	No	false
maxURLLength	integer	No	2048
noExtractBetweens	RegexPair[]	No	-
noExtractSelectors	string[]	No	-
restrictions	PropertyMatcher[]	No	-
schemes	string[]	No	[http, https, ftp]
tagAttribs	Record<string, string[]>	No	{"a":["href"],"img":["src"],"meta":["http-equiv"],"iframe":["src"],"frame":["src"]}

Property Details

charset

The character encoding to use when reading or writing the document.

commentsEnabled

When enabled, links found inside HTML comments are also extracted. By default, link extraction ignores HTML comments.

contentTypeMatcher

Restricts processing to documents whose content type matches the specified pattern. When not set, the handler applies to all content types.

extractBetweens

Regex pattern pairs (start/end) delimiting portions of a document to be included for link extraction. Only links found within the matched regions are extracted.

extractSelectors

CSS/jQuery-style selectors identifying portions of a document to be included for link extraction. Only links within the matched elements are extracted.

fieldMatcher

Matches document fields based on specified patterns and methods. Use this to restrict the handler to operate only on fields whose names satisfy the matcher criteria.

ignoreLinkData

When enabled, extra data attributes on links (such as data-* attributes) are ignored during link extraction.

ignoreNofollow

When enabled, links marked with rel="nofollow" are followed and extracted anyway, ignoring the nofollow directive.

maxURLLength

The maximum supported URL length in characters. URLs exceeding this length are ignored. Default is 2048.

noExtractBetweens

Regex pattern pairs (start/end) delimiting portions of a document to be excluded from link extraction. Links found within the matched regions are not extracted.

noExtractSelectors

CSS/jQuery-style selectors identifying portions of a document to be excluded from link extraction. Links within the matched elements are not extracted.

restrictions

Defines filter conditions that restrict which documents this committer processes. Documents not matching the restrictions are ignored by this committer.

schemes

A list of URL schemes (e.g., http, https, ftp) that are accepted or filtered.

tagAttribs

A map of HTML tag names to the list of attributes from which URLs are extracted. For example, a maps to href, img maps to src. By default includes a[href], frame[src], iframe[src], img[src], and meta[http-equiv].