UrlExtractorTransformer

Extracts unique URLs matching patterns

importerJavadoc: UrlExtractorTransformer

Notes

Extracts unique URLs matching specific patterns in plain text content and store them in a given field.

URL-matching patterns used are relatively simple. It looks for strings starting with http://, https://, or www.. The later is prefixed with https:// when encountered (to make it absolute).

The matching is case-insensitive. If you need alternate ways to detect URLs, you can use a combination of RegexTransformer, ReplaceTransformer, or create your own implementation.

Storing values in an existing field

If a target field with the same name already exists for a document, values will be added to the end of the existing value list. It is possible to change this default behavior by supplying a PropertySetter.

If no URLs are found, the target field values (if any) are left intact.

Content source

It is possible to specify a fromField as the source of the text to use instead of using the document content.

This class is typically e used as a post-parsing handler only (to ensure we are dealing with text).

Examples

The above example is used as a post-parse handler. It detects URLs in parsed PDFs and store those URLs in a field call "documentURLs".

YAML

handler:
  class: UrlExtractorTransformer
  toField: documentURLs
  fieldMatcher:
    pattern: document.contentType

JSON

{
  "handler": {
    "class": "UrlExtractorTransformer",
    "toField": "documentURLs",
    "fieldMatcher": {
      "pattern": "document.contentType"
    }
  }
}

XML

<handler>
  <class>UrlExtractorTransformer</class>
  <toField>documentURLs</toField>
  <fieldMatcher>
    <pattern>document.contentType</pattern>
  </fieldMatcher>
</handler>

Usage

Full configuration skeleton, with every option and its default

YAML

class: UrlExtractorTransformer
fieldMatcher:
  ignoreCase: false
  ignoreDiacritic: false
  matchEmpty: false
  method: BASIC
  negateMatches: false
  partial: false
  pattern: string
  replaceAll: false
  trim: false
label: string
maxReadSize: 0
onSet: APPEND
sourceCharset: string
toField: string

JSON

{
  "class": "UrlExtractorTransformer",
  "fieldMatcher": {
    "ignoreCase": false,
    "ignoreDiacritic": false,
    "matchEmpty": false,
    "method": "BASIC",
    "negateMatches": false,
    "partial": false,
    "pattern": "string",
    "replaceAll": false,
    "trim": false
  },
  "label": "string",
  "maxReadSize": 0,
  "onSet": "APPEND",
  "sourceCharset": "string",
  "toField": "string"
}

XML

<handler>
  <class>UrlExtractorTransformer</class>
  <fieldMatcher>
    <ignoreCase>false</ignoreCase>
    <ignoreDiacritic>false</ignoreDiacritic>
    <matchEmpty>false</matchEmpty>
    <method>BASIC</method>
    <negateMatches>false</negateMatches>
    <partial>false</partial>
    <pattern>string</pattern>
    <replaceAll>false</replaceAll>
    <trim>false</trim>
  </fieldMatcher>
  <label>string</label>
  <maxReadSize>0</maxReadSize>
  <onSet>APPEND</onSet>
  <sourceCharset>string</sourceCharset>
  <toField>string</toField>
</handler>

Properties

Property	Type	Required	Default
fieldMatcher	TextMatcher	No	TextMatcher
label	string	No	-
maxReadSize	integer	No	10000000
onSet	enum	No	-
sourceCharset	string	No	-
toField	string	No	-

Property Details

fieldMatcher

Matches document fields based on specified patterns and methods

label

An optional, user-supplied label for this step in the importer pipeline. It has no effect on processing — it exists purely to help identify this step, for example in large configurations or in the visual configurator.

maxReadSize

Max characters to read at once

onSet

What to do if a target field of the same name already exist.

Allowed Values

• APPEND
• PREPEND
• REPLACE
• OPTIONAL

sourceCharset

Sets the character encoding of the source document to ensure proper reading and processing.

toField

A target field name used to store the value.