TikaLinkExtractor

Extracts links from documents using Apache Tika.

crawler/webJavadoc: TikaLinkExtractor

Notes

Extracts links from documents using Apache Tika. Supports content-type restrictions, nofollow handling, and optional extraction from metadata fields in addition to document content.

Usage

YAML

class: TikaLinkExtractor
contentTypeMatcher:
  ignoreCase: false
  ignoreDiacritic: false
  matchEmpty: false
  method: BASIC
  negateMatches: false
  partial: false
  pattern: string
  replaceAll: false
  trim: false
fieldMatcher:
  ignoreCase: false
  ignoreDiacritic: false
  matchEmpty: false
  method: BASIC
  negateMatches: false
  partial: false
  pattern: string
  replaceAll: false
  trim: false
ignoreLinkData: false
ignoreNofollow: false
restrictions:
  - fieldMatcher: {}
    valueMatcher: {}

JSON

{
  "class": "TikaLinkExtractor",
  "contentTypeMatcher": {
    "ignoreCase": false,
    "ignoreDiacritic": false,
    "matchEmpty": false,
    "method": "BASIC",
    "negateMatches": false,
    "partial": false,
    "pattern": "string",
    "replaceAll": false,
    "trim": false
  },
  "fieldMatcher": {
    "ignoreCase": false,
    "ignoreDiacritic": false,
    "matchEmpty": false,
    "method": "BASIC",
    "negateMatches": false,
    "partial": false,
    "pattern": "string",
    "replaceAll": false,
    "trim": false
  },
  "ignoreLinkData": false,
  "ignoreNofollow": false,
  "restrictions": [
    {
      "fieldMatcher": {},
      "valueMatcher": {}
    }
  ]
}

XML

<tikaLinkExtractor>
  <class>TikaLinkExtractor</class>
  <contentTypeMatcher>
    <ignoreCase>false</ignoreCase>
    <ignoreDiacritic>false</ignoreDiacritic>
    <matchEmpty>false</matchEmpty>
    <method>BASIC</method>
    <negateMatches>false</negateMatches>
    <partial>false</partial>
    <pattern>string</pattern>
    <replaceAll>false</replaceAll>
    <trim>false</trim>
  </contentTypeMatcher>
  <fieldMatcher>
    <ignoreCase>false</ignoreCase>
    <ignoreDiacritic>false</ignoreDiacritic>
    <matchEmpty>false</matchEmpty>
    <method>BASIC</method>
    <negateMatches>false</negateMatches>
    <partial>false</partial>
    <pattern>string</pattern>
    <replaceAll>false</replaceAll>
    <trim>false</trim>
  </fieldMatcher>
  <ignoreLinkData>false</ignoreLinkData>
  <ignoreNofollow>false</ignoreNofollow>
  <restrictions>
    <restriction>
      <fieldMatcher/>
      <valueMatcher/>
    </restriction>
  </restrictions>
</tikaLinkExtractor>

Properties

Property	Type	Required	Default
contentTypeMatcher	TextMatcher	No	TextMatcher
fieldMatcher	TextMatcher	No	TextMatcher
ignoreLinkData	boolean	No	false
ignoreNofollow	boolean	No	false
restrictions	PropertyMatcher[]	No	-

Property Details

contentTypeMatcher

Restricts processing to documents whose content type matches the specified pattern. When not set, the handler applies to all content types.

fieldMatcher

Matches document fields based on specified patterns and methods. Use this to restrict the handler to operate only on fields whose names satisfy the matcher criteria.

ignoreLinkData

When enabled, extra data attributes on links (such as data-* attributes) are ignored during link extraction.

ignoreNofollow

When enabled, links marked with rel="nofollow" are followed and extracted anyway, ignoring the nofollow directive.

restrictions

Defines filter conditions that restrict which documents this committer processes. Documents not matching the restrictions are ignored by this committer.