RegexTransformer

Extracts field names and their values with regular expression

importerJavadoc: RegexTransformer

Notes

This is done by using match groups in your regular expressions (parenthesis). For each pattern you define, you can specify which match group hold the field name(fieldGroup) and which one holds the value(valueGroup). Specifying a field match group is optional if a field is provided. If no match groups are specified, a toField is expected.

If fieldMatcher is specified, it will use content from matching fields and storing all text extracted into the target field, multi-value. Else, the document content is used.

Storing values in an existing field

If a target field with the same name already exists for a document, values will be added to the end of the existing value list. It is possible to change this default behavior by supplying a PropertySetter.

This class can be used as a pre-parsing handler on text documents only or a post-parsing handler.

Examples

The first pattern below extracts what look like email addresses into an email field (a simplified regex). The second extracts field names and values from the label and value cells of an HTML table.

YAML

handler:
  class: RegexTransformer
  patterns:
    - toField: emails
      regex:
        pattern: ([a-zA-Z0-9._-]+@[a-zA-Z0-9._-]+\[a-zA-Z0-9_-]+)
    - fieldGroup: 1
      valueGroup: 2
      regex:
        pattern: <![CDATA[<tr><td class="label">(.*?)</td><td
          class="value">(.*?)</td></tr>]]>

JSON

{
  "handler": {
    "class": "RegexTransformer",
    "patterns": [
      {
        "toField": "emails",
        "regex": {
          "pattern": "([a-zA-Z0-9._-]+@[a-zA-Z0-9._-]+\\[a-zA-Z0-9_-]+)"
        }
      },
      {
        "fieldGroup": 1,
        "valueGroup": 2,
        "regex": {
          "pattern": "<![CDATA[<tr><td class=\"label\">(.*?)</td><td class=\"value\">(.*?)</td></tr>]]>"
        }
      }
    ]
  }
}

XML

<handler>
  <class>RegexTransformer</class>
  <patterns>
    <pattern>
      <toField>emails</toField>
      <regex>
        <pattern>([a-zA-Z0-9._-]+@[a-zA-Z0-9._-]+[a-zA-Z0-9_-]+)</pattern>
      </regex>
    </pattern>
    <pattern>
      <fieldGroup>1</fieldGroup>
      <valueGroup>2</valueGroup>
      <regex>
        <pattern><![CDATA[<tr><td class="label">(.*?)</td><td class="value">(.*?)</td></tr>]]></pattern>
      </regex>
    </pattern>
  </patterns>
</handler>

Usage

Full configuration skeleton, with every option and its default

YAML

class: RegexTransformer
fieldMatcher:
  ignoreCase: false
  ignoreDiacritic: false
  matchEmpty: false
  method: BASIC
  negateMatches: false
  partial: false
  pattern: string
  replaceAll: false
  trim: false
label: string
maxReadSize: 0
patterns:
  - fieldGroup: 0
    onSet: APPEND
    regex: {}
    toField: string
    valueGroup: 0
sourceCharset: string

JSON

{
  "class": "RegexTransformer",
  "fieldMatcher": {
    "ignoreCase": false,
    "ignoreDiacritic": false,
    "matchEmpty": false,
    "method": "BASIC",
    "negateMatches": false,
    "partial": false,
    "pattern": "string",
    "replaceAll": false,
    "trim": false
  },
  "label": "string",
  "maxReadSize": 0,
  "patterns": [
    {
      "fieldGroup": 0,
      "onSet": "APPEND",
      "regex": {},
      "toField": "string",
      "valueGroup": 0
    }
  ],
  "sourceCharset": "string"
}

XML

<handler>
  <class>RegexTransformer</class>
  <fieldMatcher>
    <ignoreCase>false</ignoreCase>
    <ignoreDiacritic>false</ignoreDiacritic>
    <matchEmpty>false</matchEmpty>
    <method>BASIC</method>
    <negateMatches>false</negateMatches>
    <partial>false</partial>
    <pattern>string</pattern>
    <replaceAll>false</replaceAll>
    <trim>false</trim>
  </fieldMatcher>
  <label>string</label>
  <maxReadSize>0</maxReadSize>
  <patterns>
    <pattern>
      <fieldGroup>0</fieldGroup>
      <onSet>APPEND</onSet>
      <regex/>
      <toField>string</toField>
      <valueGroup>0</valueGroup>
    </pattern>
  </patterns>
  <sourceCharset>string</sourceCharset>
</handler>

Properties

Property	Type	Required	Default
fieldMatcher	TextMatcher	No	TextMatcher
label	string	No	-
maxReadSize	integer	No	10000000
patterns	RegexFieldValueExtractor[]	No	-
sourceCharset	string	No	-

Property Details

fieldMatcher

Matches document fields based on specified patterns and methods

label

An optional, user-supplied label for this step in the importer pipeline. It has no effect on processing — it exists purely to help identify this step, for example in large configurations or in the visual configurator.

maxReadSize

Max characters to read at once

patterns

Regular expression list of patterns to match.

sourceCharset

Sets the character encoding of the source document to ensure proper reading and processing.