RegexLinkExtractor

Extracts links from documents using regular expression patterns.

crawler/webJavadoc: RegexLinkExtractor

Notes

Extracts links from documents using regular expressions. Each pattern can include a match group to capture the URL and an optional replacement expression to construct the final URL from the match. Supports content-type restrictions and URL length limits.

Examples

The following example extracts page "ids" contained in square brackets and add them to a custom URL.

YAML

class: RegexLinkExtractor
patterns:
  - match: \[(\d+)\]
    replace: http://www.example.com/page?id=$1

JSON

{
  "class": "RegexLinkExtractor",
  "patterns": [
    {
      "match": "\\[(\\d+)\\]",
      "replace": "http://www.example.com/page?id=$1"
    }
  ]
}

XML

<class>RegexLinkExtractor</class>
<patterns>
  <pattern>
    <match>[(d+)]</match>
    <replace>http://www.example.com/page?id=$1</replace>
  </pattern>
</patterns>

Usage

Full configuration skeleton, with every option and its default

YAML

class: RegexLinkExtractor
charset: string
contentTypeMatcher:
  ignoreCase: false
  ignoreDiacritic: false
  matchEmpty: false
  method: BASIC
  negateMatches: false
  partial: false
  pattern: string
  replaceAll: false
  trim: false
fieldMatcher:
  ignoreCase: false
  ignoreDiacritic: false
  matchEmpty: false
  method: BASIC
  negateMatches: false
  partial: false
  pattern: string
  replaceAll: false
  trim: false
maxUrlLength: 0
patterns:
  - match: string
    replace: string
restrictions:
  - fieldMatcher: {}
    valueMatcher: {}

JSON

{
  "class": "RegexLinkExtractor",
  "charset": "string",
  "contentTypeMatcher": {
    "ignoreCase": false,
    "ignoreDiacritic": false,
    "matchEmpty": false,
    "method": "BASIC",
    "negateMatches": false,
    "partial": false,
    "pattern": "string",
    "replaceAll": false,
    "trim": false
  },
  "fieldMatcher": {
    "ignoreCase": false,
    "ignoreDiacritic": false,
    "matchEmpty": false,
    "method": "BASIC",
    "negateMatches": false,
    "partial": false,
    "pattern": "string",
    "replaceAll": false,
    "trim": false
  },
  "maxUrlLength": 0,
  "patterns": [
    {
      "match": "string",
      "replace": "string"
    }
  ],
  "restrictions": [
    {
      "fieldMatcher": {},
      "valueMatcher": {}
    }
  ]
}

XML

<regexLinkExtractor>
  <class>RegexLinkExtractor</class>
  <charset>string</charset>
  <contentTypeMatcher>
    <ignoreCase>false</ignoreCase>
    <ignoreDiacritic>false</ignoreDiacritic>
    <matchEmpty>false</matchEmpty>
    <method>BASIC</method>
    <negateMatches>false</negateMatches>
    <partial>false</partial>
    <pattern>string</pattern>
    <replaceAll>false</replaceAll>
    <trim>false</trim>
  </contentTypeMatcher>
  <fieldMatcher>
    <ignoreCase>false</ignoreCase>
    <ignoreDiacritic>false</ignoreDiacritic>
    <matchEmpty>false</matchEmpty>
    <method>BASIC</method>
    <negateMatches>false</negateMatches>
    <partial>false</partial>
    <pattern>string</pattern>
    <replaceAll>false</replaceAll>
    <trim>false</trim>
  </fieldMatcher>
  <maxUrlLength>0</maxUrlLength>
  <patterns>
    <pattern>
      <match>string</match>
      <replace>string</replace>
    </pattern>
  </patterns>
  <restrictions>
    <restriction>
      <fieldMatcher/>
      <valueMatcher/>
    </restriction>
  </restrictions>
</regexLinkExtractor>

Properties

Property	Type	Required	Default
charset	string	No	-
contentTypeMatcher	TextMatcher	No	TextMatcher
fieldMatcher	TextMatcher	No	TextMatcher
maxUrlLength	integer	No	2048
patterns	ExtractionPattern[]	No	-
restrictions	PropertyMatcher[]	No	-

Property Details

charset

The character encoding to use when reading or writing the document.

contentTypeMatcher

Restricts processing to documents whose content type matches the specified pattern. When not set, the handler applies to all content types.

fieldMatcher

Matches document fields based on specified patterns and methods. Use this to restrict the handler to operate only on fields whose names satisfy the matcher criteria.

maxUrlLength

The maximum supported URL length in characters. URLs exceeding this length are ignored. Default is 2048.

patterns

A list of extraction patterns, each consisting of a match regular expression and an optional replace expression for constructing the final URL from captured groups.

restrictions

Defines filter conditions that restrict which documents this committer processes. Documents not matching the restrictions are ignored by this committer.