TextStatisticsTransformer

Document content analysis and statistical metadata generation.

importerJavadoc: TextStatisticsTransformer

Notes

This feature analyzes the content of the supplied document and adds statistical information as metadata fields. Default behavior provide the statistics about the content. Refer to the following for the new metadata fields to be created along with their description.

Statistic fields

Field name	Description
document.stat.characterCount	Total number of characters (excluding carriage returns/line feed).
document.stat.wordCount	Total number of words.
document.stat.sentenceCount	Total number of sentences.
document.stat.paragraphCount	Total number of paragraphs.
document.stat.averageWordCharacterCount	Average number of characters in every word.
document.stat.averageSentenceCharacterCount	Average number of characters in sentences (including non-word characters, such as spaces or slashes).
document.stat.averageSentenceWordCount	Average number of words per sentence.
document.stat.averageParagraphCharacterCount	Average number of characters in paragraphs (including non-word characters, such as spaces or slashes).
document.stat.averageParagraphSentenceCount	Average number of sentences per paragraph.
document.stat.averageParagraphWordCount	Average number of words per paragraph.

You can specify a field matcher to obtain statistics about matching fields instead. When you do so, the field name will be inserted in the above names, right after document.stat.. E.g.: document.stat.myfield.characterCount

Can be used both as a pre-parse (text-only) or post-parse handler.

Examples

The following example create statistics from the value of a field called "statistics".

YAML

handler:
  class: TextStatisticsTransformer
  fieldMatcher:
    pattern: statistics

JSON

{
  "handler": {
    "class": "TextStatisticsTransformer",
    "fieldMatcher": {
      "pattern": "statistics"
    }
  }
}

XML

<handler>
  <class>TextStatisticsTransformer</class>
  <fieldMatcher>
    <pattern>statistics</pattern>
  </fieldMatcher>
</handler>

Usage

Full configuration skeleton, with every option and its default

YAML

class: TextStatisticsTransformer
fieldMatcher:
  ignoreCase: false
  ignoreDiacritic: false
  matchEmpty: false
  method: BASIC
  negateMatches: false
  partial: false
  pattern: string
  replaceAll: false
  trim: false
label: string
sourceCharset: string

JSON

{
  "class": "TextStatisticsTransformer",
  "fieldMatcher": {
    "ignoreCase": false,
    "ignoreDiacritic": false,
    "matchEmpty": false,
    "method": "BASIC",
    "negateMatches": false,
    "partial": false,
    "pattern": "string",
    "replaceAll": false,
    "trim": false
  },
  "label": "string",
  "sourceCharset": "string"
}

XML

<handler>
  <class>TextStatisticsTransformer</class>
  <fieldMatcher>
    <ignoreCase>false</ignoreCase>
    <ignoreDiacritic>false</ignoreDiacritic>
    <matchEmpty>false</matchEmpty>
    <method>BASIC</method>
    <negateMatches>false</negateMatches>
    <partial>false</partial>
    <pattern>string</pattern>
    <replaceAll>false</replaceAll>
    <trim>false</trim>
  </fieldMatcher>
  <label>string</label>
  <sourceCharset>string</sourceCharset>
</handler>

Properties

Property	Type	Required	Default
fieldMatcher	TextMatcher	No	TextMatcher
label	string	No	-
sourceCharset	string	No	-

Property Details

fieldMatcher

Matches document fields based on specified patterns and methods

label

An optional, user-supplied label for this step in the importer pipeline. It has no effect on processing — it exists purely to help identify this step, for example in large configurations or in the visual configurator.

sourceCharset

Sets the character encoding of the source document to ensure proper reading and processing.