CharsetTransformer

Converts a document's character encoding.

importerJavadoc: CharsetTransformer

Notes

Converts one or more field values (if needed) from a source character encoding (charset) to a target one. Both the source and target character encodings are optional. If no source character encoding is explicitly provided, it first tries to detect the encoding of the field values before converting them to the target encoding. If the source character encoding cannot be established, the content encoding will remain unchanged. When no target character encoding is specified, UTF-8 is assumed.

Should I use this transformer?

Before using this transformer, you need to know the parsing of documents by the importer (using the default document parser factory) will try to convert and return fields as UTF-8 (for most, if not all content-types). If UTF-8 is your desired target, it only make sense to use this transformer before parsing handler (for text content-types only) when it is important to work with a specific character encoding before parsing. If on the other hand you wish to convert to a character encoding to a target different than UTF-8, you can use this transformer after parsing handler to do so.

Conversion is not flawless

Because character encoding detection is not always accurate and because documents sometime mix different encoding, there is no guarantee this class will handle ALL character encoding conversions properly.

Examples

The following example converts the characters of a "description" field from "ISO-8859-1" to "UTF-8".

YAML

handler:
  class: CharsetTransformer
  sourceCharset: ISO-8859-1
  targetCharset: UTF-8
  fieldMatcher:
    pattern: description

JSON

{
  "handler": {
    "class": "CharsetTransformer",
    "sourceCharset": "ISO-8859-1",
    "targetCharset": "UTF-8",
    "fieldMatcher": {
      "pattern": "description"
    }
  }
}

XML

<handler>
  <class>CharsetTransformer</class>
  <sourceCharset>ISO-8859-1</sourceCharset>
  <targetCharset>UTF-8</targetCharset>
  <fieldMatcher>
    <pattern>description</pattern>
  </fieldMatcher>
</handler>

Usage

Full configuration skeleton, with every option and its default

YAML

class: CharsetTransformer
fieldMatcher:
  ignoreCase: false
  ignoreDiacritic: false
  matchEmpty: false
  method: BASIC
  negateMatches: false
  partial: false
  pattern: string
  replaceAll: false
  trim: false
label: string
sourceCharset: string
targetCharset: string

JSON

{
  "class": "CharsetTransformer",
  "fieldMatcher": {
    "ignoreCase": false,
    "ignoreDiacritic": false,
    "matchEmpty": false,
    "method": "BASIC",
    "negateMatches": false,
    "partial": false,
    "pattern": "string",
    "replaceAll": false,
    "trim": false
  },
  "label": "string",
  "sourceCharset": "string",
  "targetCharset": "string"
}

XML

<handler>
  <class>CharsetTransformer</class>
  <fieldMatcher>
    <ignoreCase>false</ignoreCase>
    <ignoreDiacritic>false</ignoreDiacritic>
    <matchEmpty>false</matchEmpty>
    <method>BASIC</method>
    <negateMatches>false</negateMatches>
    <partial>false</partial>
    <pattern>string</pattern>
    <replaceAll>false</replaceAll>
    <trim>false</trim>
  </fieldMatcher>
  <label>string</label>
  <sourceCharset>string</sourceCharset>
  <targetCharset>string</targetCharset>
</handler>

Properties

Property	Type	Required	Default
fieldMatcher	TextMatcher	No	TextMatcher
label	string	No	-
sourceCharset	string	No	-
targetCharset	string	No	UTF_8

Property Details

fieldMatcher

Matches document fields based on specified patterns and methods. Use this to restrict the handler to operate only on fields whose names satisfy the matcher criteria.

label

An optional, user-supplied label for this step in the importer pipeline. It has no effect on processing — it exists purely to help identify this step, for example in large configurations or in the visual configurator.

sourceCharset

Sets the character encoding of the source document to ensure proper reading and processing.

targetCharset

Defines the character encoding to be applied to the document after processing, ensuring correct output formatting.