CollapseRepeatingTransformer

Reduces specified consecutive characters or strings

importerJavadoc: CollapseRepeatingTransformer

Notes

This feature reduces repeated consecutive characters in a text to a single instance or a specified number of occurrences. This transformer is useful for normalizing text by removing unnecessary repetitions, such as excessive punctuation or repeated whitespace. For instance, it can reduce "aaaabbb" to "ab" or limit it to a specific pattern like "aa". It operates on text fields based on specified conditions and is often applied during preprocessing to clean and normalize data for further processing.

In addition to regular characters, you can specify these special characters in reduce as:

• \r (carriage returns)
• \n (line feed)
• \t (tab)
• \s (space)

For more advanced replacement needs, consider using ReplaceTransformer instead.

Parser Handler Compatibility

This functionality can be applied as either a pre-parse or post-parse handler.

Examples

The following example reduces multiple line feed and tab into a single one.

YAML

handler:
  class: CollapseRepeatingTransformer
  strings:
    - |+
      
    - "\t"

JSON

{
  "handler": {
    "class": "CollapseRepeatingTransformer",
    "strings": [
      "\n",
      "\t"
    ]
  }
}

XML

<handler>
  <class>CollapseRepeatingTransformer</class>
  <strings>
    <string></string>
    <string></string>
  </strings>
</handler>

Usage

Full configuration skeleton, with every option and its default

YAML

class: CollapseRepeatingTransformer
fieldMatcher:
  ignoreCase: false
  ignoreDiacritic: false
  matchEmpty: false
  method: BASIC
  negateMatches: false
  partial: false
  pattern: string
  replaceAll: false
  trim: false
ignoreCase: false
label: string
maxReadSize: 0
sourceCharset: string
strings:
  - string

JSON

{
  "class": "CollapseRepeatingTransformer",
  "fieldMatcher": {
    "ignoreCase": false,
    "ignoreDiacritic": false,
    "matchEmpty": false,
    "method": "BASIC",
    "negateMatches": false,
    "partial": false,
    "pattern": "string",
    "replaceAll": false,
    "trim": false
  },
  "ignoreCase": false,
  "label": "string",
  "maxReadSize": 0,
  "sourceCharset": "string",
  "strings": [
    "string"
  ]
}

XML

<handler>
  <class>CollapseRepeatingTransformer</class>
  <fieldMatcher>
    <ignoreCase>false</ignoreCase>
    <ignoreDiacritic>false</ignoreDiacritic>
    <matchEmpty>false</matchEmpty>
    <method>BASIC</method>
    <negateMatches>false</negateMatches>
    <partial>false</partial>
    <pattern>string</pattern>
    <replaceAll>false</replaceAll>
    <trim>false</trim>
  </fieldMatcher>
  <ignoreCase>false</ignoreCase>
  <label>string</label>
  <maxReadSize>0</maxReadSize>
  <sourceCharset>string</sourceCharset>
  <strings>
    <string>string</string>
  </strings>
</handler>

Properties

Property	Type	Required	Default
fieldMatcher	TextMatcher	No	TextMatcher
ignoreCase	boolean	No	false
label	string	No	-
maxReadSize	integer	No	10000000
sourceCharset	string	No	-
strings	string[]	No	-

Property Details

fieldMatcher

Matches document fields based on specified patterns and methods. Use this to restrict the handler to operate only on fields whose names satisfy the matcher criteria.

ignoreCase

When enabled, matching is performed without regard to character case.

label

An optional, user-supplied label for this step in the importer pipeline. It has no effect on processing — it exists purely to help identify this step, for example in large configurations or in the visual configurator.

maxReadSize

The maximum number of characters to read from the document content at once. Use this to limit memory usage when processing large documents.

sourceCharset

Sets the character encoding of the source document to ensure proper reading and processing.

strings

The list of characters or strings whose consecutive repetitions should be collapsed to a single occurrence.