XmlFeedLinkExtractor

Extracts links from RSS and Atom XML feed documents.

crawler/webJavadoc: XmlFeedLinkExtractor

Notes

Extracts links from RSS and Atom XML feed documents by parsing standard feed elements. Supports content-type restrictions to limit extraction to feed-type documents.

Examples

The following example ensures this link extractor should only apply on documents that have their URL ending with "rss" (in addition to the default content types filtering).

YAML

class: XmlFeedLinkExtractor
restrictions:
  - fieldMatcher:
      pattern: document.reference
    valueMatcher:
      method: REGEX
      pattern: .*rss$

JSON

{
  "class": "XmlFeedLinkExtractor",
  "restrictions": [
    {
      "fieldMatcher": {
        "pattern": "document.reference"
      },
      "valueMatcher": {
        "method": "REGEX",
        "pattern": ".*rss$"
      }
    }
  ]
}

XML

<class>XmlFeedLinkExtractor</class>
<restrictions>
  <restriction>
    <fieldMatcher>
      <pattern>document.reference</pattern>
    </fieldMatcher>
    <valueMatcher>
      <method>REGEX</method>
      <pattern>.*rss$</pattern>
    </valueMatcher>
  </restriction>
</restrictions>

Usage

Full configuration skeleton, with every option and its default

YAML

class: XmlFeedLinkExtractor
contentTypeMatcher:
  ignoreCase: false
  ignoreDiacritic: false
  matchEmpty: false
  method: BASIC
  negateMatches: false
  partial: false
  pattern: string
  replaceAll: false
  trim: false
fieldMatcher:
  ignoreCase: false
  ignoreDiacritic: false
  matchEmpty: false
  method: BASIC
  negateMatches: false
  partial: false
  pattern: string
  replaceAll: false
  trim: false
restrictions:
  - fieldMatcher: {}
    valueMatcher: {}

JSON

{
  "class": "XmlFeedLinkExtractor",
  "contentTypeMatcher": {
    "ignoreCase": false,
    "ignoreDiacritic": false,
    "matchEmpty": false,
    "method": "BASIC",
    "negateMatches": false,
    "partial": false,
    "pattern": "string",
    "replaceAll": false,
    "trim": false
  },
  "fieldMatcher": {
    "ignoreCase": false,
    "ignoreDiacritic": false,
    "matchEmpty": false,
    "method": "BASIC",
    "negateMatches": false,
    "partial": false,
    "pattern": "string",
    "replaceAll": false,
    "trim": false
  },
  "restrictions": [
    {
      "fieldMatcher": {},
      "valueMatcher": {}
    }
  ]
}

XML

<xmlFeedLinkExtractor>
  <class>XmlFeedLinkExtractor</class>
  <contentTypeMatcher>
    <ignoreCase>false</ignoreCase>
    <ignoreDiacritic>false</ignoreDiacritic>
    <matchEmpty>false</matchEmpty>
    <method>BASIC</method>
    <negateMatches>false</negateMatches>
    <partial>false</partial>
    <pattern>string</pattern>
    <replaceAll>false</replaceAll>
    <trim>false</trim>
  </contentTypeMatcher>
  <fieldMatcher>
    <ignoreCase>false</ignoreCase>
    <ignoreDiacritic>false</ignoreDiacritic>
    <matchEmpty>false</matchEmpty>
    <method>BASIC</method>
    <negateMatches>false</negateMatches>
    <partial>false</partial>
    <pattern>string</pattern>
    <replaceAll>false</replaceAll>
    <trim>false</trim>
  </fieldMatcher>
  <restrictions>
    <restriction>
      <fieldMatcher/>
      <valueMatcher/>
    </restriction>
  </restrictions>
</xmlFeedLinkExtractor>

Properties

Property	Type	Required	Default
contentTypeMatcher	TextMatcher	No	TextMatcher
fieldMatcher	TextMatcher	No	TextMatcher
restrictions	PropertyMatcher[]	No	-

Property Details

contentTypeMatcher

Restricts processing to documents whose content type matches the specified pattern. When not set, the handler applies to all content types.

fieldMatcher

Matches document fields based on specified patterns and methods. Use this to restrict the handler to operate only on fields whose names satisfy the matcher criteria.

restrictions

Defines filter conditions that restrict which documents this committer processes. Documents not matching the restrictions are ignored by this committer.