TextBetweenTransformer

Extracts and add values found between a matching start and end strings to a document metadata field

importerJavadoc: TextBetweenTransformer

Notes

The matching string end-points are defined in pairs and multiple ones can be specified at once. The field specified for a pair of end-points is considered a multi-value field.

If fieldMatcher is specified, it will use content from matching fields and storing all text extracted into the target field, multi-value. Else, the document content is used.

Storing values in an existing field

If a target field with the same name already exists for a document, values will be added to the end of the existing value list. It is possible to change this default behavior by supplying a PropertySetter.

This class can be used as a pre-parsing handler on text documents only or a post-parsing handler.

Examples

The above example extract the content between OPEN and CLOSE strings, excluding these strings, and store it in a content field.

YAML

handler:
  class: TextBetweenTransformer
  operations:
    - toField: content
      startMatcher:
        pattern: OPEN
      endMatcher:
        pattern: CLOSE

JSON

{
  "handler": {
    "class": "TextBetweenTransformer",
    "operations": [
      {
        "toField": "content",
        "startMatcher": {
          "pattern": "OPEN"
        },
        "endMatcher": {
          "pattern": "CLOSE"
        }
      }
    ]
  }
}

XML

<handler>
  <class>TextBetweenTransformer</class>
  <operations>
    <op>
      <toField>content</toField>
      <startMatcher>
        <pattern>OPEN</pattern>
      </startMatcher>
      <endMatcher>
        <pattern>CLOSE</pattern>
      </endMatcher>
    </op>
  </operations>
</handler>

Usage

Full configuration skeleton, with every option and its default

YAML

class: TextBetweenTransformer
label: string
maxReadSize: 0
operations:
  - endMatcher: {}
    fieldMatcher: {}
    inclusive: false
    onSet: APPEND
    startMatcher: {}
    toField: string
sourceCharset: string

JSON

{
  "class": "TextBetweenTransformer",
  "label": "string",
  "maxReadSize": 0,
  "operations": [
    {
      "endMatcher": {},
      "fieldMatcher": {},
      "inclusive": false,
      "onSet": "APPEND",
      "startMatcher": {},
      "toField": "string"
    }
  ],
  "sourceCharset": "string"
}

XML

<handler>
  <class>TextBetweenTransformer</class>
  <label>string</label>
  <maxReadSize>0</maxReadSize>
  <operations>
    <op>
      <endMatcher/>
      <fieldMatcher/>
      <inclusive>false</inclusive>
      <onSet>APPEND</onSet>
      <startMatcher/>
      <toField>string</toField>
    </op>
  </operations>
  <sourceCharset>string</sourceCharset>
</handler>

Properties

Property	Type	Required	Default
label	string	No	-
maxReadSize	integer	No	10000000
operations	TextBetweenOperation[]	No	-
sourceCharset	string	No	-

Property Details

label

An optional, user-supplied label for this step in the importer pipeline. It has no effect on processing — it exists purely to help identify this step, for example in large configurations or in the visual configurator.

maxReadSize

Max characters to read at once

operations

Allows defining multiple operations as a list.

sourceCharset

Sets the character encoding of the source document to ensure proper reading and processing.