DomLinkExtractor

Extracts links from HTML/XML documents using CSS/jQuery-style selectors.

crawler/webJavadoc: DomLinkExtractor

Notes

Extracts links from HTML, XHTML, or XML documents using CSS/jQuery-style selectors. Supports custom link selectors, extraction of specific element attributes, nofollow handling, and content-type restrictions.

Examples

The following example represents the default link extraction selectors with the addition of one for URLs found in custom element attributes named data-myurl.

YAML

class: DomLinkExtractor
linkSelectors:
  - extract: attr(href)
    selector: a[href]
  - extract: attr(src)
    selector: "[src]"
  - extract: attr(href)
    selector: link[href]
  - extract: attr(content)
    selector: meta[http-equiv='refresh']
  - extract: attr(data-myurl)
    selector: "[data-myurl]"

JSON

{
  "class": "DomLinkExtractor",
  "linkSelectors": [
    {
      "extract": "attr(href)",
      "selector": "a[href]"
    },
    {
      "extract": "attr(src)",
      "selector": "[src]"
    },
    {
      "extract": "attr(href)",
      "selector": "link[href]"
    },
    {
      "extract": "attr(content)",
      "selector": "meta[http-equiv='refresh']"
    },
    {
      "extract": "attr(data-myurl)",
      "selector": "[data-myurl]"
    }
  ]
}

XML

<class>DomLinkExtractor</class>
<linkSelectors>
  <linkSelector>
    <extract>attr(href)</extract>
    <selector>a[href]</selector>
  </linkSelector>
  <linkSelector>
    <extract>attr(src)</extract>
    <selector>[src]</selector>
  </linkSelector>
  <linkSelector>
    <extract>attr(href)</extract>
    <selector>link[href]</selector>
  </linkSelector>
  <linkSelector>
    <extract>attr(content)</extract>
    <selector>meta[http-equiv='refresh']</selector>
  </linkSelector>
  <linkSelector>
    <extract>attr(data-myurl)</extract>
    <selector>[data-myurl]</selector>
  </linkSelector>
</linkSelectors>

Usage

Full configuration skeleton, with every option and its default

YAML

class: DomLinkExtractor
charset: string
contentTypeMatcher:
  ignoreCase: false
  ignoreDiacritic: false
  matchEmpty: false
  method: BASIC
  negateMatches: false
  partial: false
  pattern: string
  replaceAll: false
  trim: false
extractSelectors:
  - string
fieldMatcher:
  ignoreCase: false
  ignoreDiacritic: false
  matchEmpty: false
  method: BASIC
  negateMatches: false
  partial: false
  pattern: string
  replaceAll: false
  trim: false
ignoreLinkData: false
ignoreNofollow: false
linkSelectors:
  - extract: string
    selector: string
noExtractSelectors:
  - string
parser: string
restrictions:
  - fieldMatcher: {}
    valueMatcher: {}
schemes:
  - string

JSON

{
  "class": "DomLinkExtractor",
  "charset": "string",
  "contentTypeMatcher": {
    "ignoreCase": false,
    "ignoreDiacritic": false,
    "matchEmpty": false,
    "method": "BASIC",
    "negateMatches": false,
    "partial": false,
    "pattern": "string",
    "replaceAll": false,
    "trim": false
  },
  "extractSelectors": [
    "string"
  ],
  "fieldMatcher": {
    "ignoreCase": false,
    "ignoreDiacritic": false,
    "matchEmpty": false,
    "method": "BASIC",
    "negateMatches": false,
    "partial": false,
    "pattern": "string",
    "replaceAll": false,
    "trim": false
  },
  "ignoreLinkData": false,
  "ignoreNofollow": false,
  "linkSelectors": [
    {
      "extract": "string",
      "selector": "string"
    }
  ],
  "noExtractSelectors": [
    "string"
  ],
  "parser": "string",
  "restrictions": [
    {
      "fieldMatcher": {},
      "valueMatcher": {}
    }
  ],
  "schemes": [
    "string"
  ]
}

XML

<domLinkExtractor>
  <class>DomLinkExtractor</class>
  <charset>string</charset>
  <contentTypeMatcher>
    <ignoreCase>false</ignoreCase>
    <ignoreDiacritic>false</ignoreDiacritic>
    <matchEmpty>false</matchEmpty>
    <method>BASIC</method>
    <negateMatches>false</negateMatches>
    <partial>false</partial>
    <pattern>string</pattern>
    <replaceAll>false</replaceAll>
    <trim>false</trim>
  </contentTypeMatcher>
  <extractSelectors>
    <extractSelector>string</extractSelector>
  </extractSelectors>
  <fieldMatcher>
    <ignoreCase>false</ignoreCase>
    <ignoreDiacritic>false</ignoreDiacritic>
    <matchEmpty>false</matchEmpty>
    <method>BASIC</method>
    <negateMatches>false</negateMatches>
    <partial>false</partial>
    <pattern>string</pattern>
    <replaceAll>false</replaceAll>
    <trim>false</trim>
  </fieldMatcher>
  <ignoreLinkData>false</ignoreLinkData>
  <ignoreNofollow>false</ignoreNofollow>
  <linkSelectors>
    <linkSelector>
      <extract>string</extract>
      <selector>string</selector>
    </linkSelector>
  </linkSelectors>
  <noExtractSelectors>
    <noExtractSelector>string</noExtractSelector>
  </noExtractSelectors>
  <parser>string</parser>
  <restrictions>
    <restriction>
      <fieldMatcher/>
      <valueMatcher/>
    </restriction>
  </restrictions>
  <schemes>
    <scheme>string</scheme>
  </schemes>
</domLinkExtractor>

Properties

Property	Type	Required	Default
charset	string	No	-
contentTypeMatcher	TextMatcher	No	TextMatcher
extractSelectors	string[]	No	-
fieldMatcher	TextMatcher	No	TextMatcher
ignoreLinkData	boolean	No	false
ignoreNofollow	boolean	No	false
linkSelectors	LinkSelector[]	No	[LinkSelector, LinkSelector, LinkSelector, LinkSelector]
noExtractSelectors	string[]	No	-
parser	string	No	html
restrictions	PropertyMatcher[]	No	-
schemes	string[]	No	[http, https, ftp]

Property Details

charset

The character encoding to use when reading or writing the document.

contentTypeMatcher

Restricts processing to documents whose content type matches the specified pattern. When not set, the handler applies to all content types.

extractSelectors

CSS/jQuery-style selectors identifying portions of a document to be included for link extraction. Only links within the matched elements are considered by the link selectors.

fieldMatcher

Matches document fields based on specified patterns and methods. Use this to restrict the handler to operate only on fields whose names satisfy the matcher criteria.

ignoreLinkData

When enabled, extra data attributes on links (such as data-* attributes) are ignored during link extraction.

ignoreNofollow

When enabled, links marked with rel="nofollow" are followed and extracted anyway, ignoring the nofollow directive.

linkSelectors

CSS/jQuery-style selectors paired with extraction expressions that identify which elements to scan and what data to extract as a URL (e.g., attr(href) to extract an attribute value, or text for the element's text content). Defaults include selectors for a[href], [src], link[href], and meta[http-equiv='refresh'].

noExtractSelectors

CSS/jQuery-style selectors identifying portions of a document to be excluded from link extraction. Link selectors are not applied to elements matching these selectors.

parser

Specifies whether to use the HTML or XML parser when reading document content into a DOM tree.

restrictions

Defines filter conditions that restrict which documents this committer processes. Documents not matching the restrictions are ignored by this committer.

schemes

A list of URL schemes (e.g., http, https, ftp) that are accepted or filtered.