TitleGeneratorTransformer

Title Generation from Document Content

importerJavadoc: TitleGeneratorTransformer

Notes

Attempts to generate a title from the document content (default) or a specified metadata field. It does not consider the document format/structure nor does it weight some terms more than others. For instance, it would not consider text found in <H1> tags more importantly than other text in HTML documents.

If isDetectHeading() returns true, this handler will check if the content starts with a stand-alone, single-sentence line (which is assumed to be the actual title). That is, a line of text with only one sentence in it, followed by one or more new line characters. To help eliminate cases where such sentence are inappropriate, you can specify a minimum and maximum number of characters that first line should have with setDetectHeadingMinLength(int) and setDetectHeadingMaxLength(int) (e.g. to ignore "Page 1" text and the like).

Unless a target field name is provided, the default field name where the title will be stored is document.generatedTitle.

Storing values in an existing field

If a target field with the same name already exists for a document, values will be added to the end of the existing value list. It is possible to change this default behavior by supplying a PropertySetter.

If it cannot generate a title, it will fall-back to retrieving the first sentence from the text.

The generated title length is limited to 150 characters by default. You can change that limit by using setTitleMaxLength(int). Text larger than the max limit will be truncated and three dots will be added in square brackets ([...]). To remove the limit, use -1 (or constant UNLIMITED_TITLE_LENGTH).

This class should be used as a post-parsing handler only (or otherwise on unformatted text).

The algorithm to detect titles is quite basic. It uses a generic statistics-based approach to weight each sentences up to a certain amount, and simply returns the sentence with the highest attributed weight given a minimum threshold has been met. You are strongly encouraged to use a more sophisticated summarization engine if you want more accurate titles generated.

Max read size

Only the beginning of the content is analyzed, as set by maxReadSize. This handler is not optimized for large content, so raising that limit a long way can cost real performance on big files.

Examples

The follow example will check if the first line looks like a title and if not, it will store the first sentence, up to 200 characters, in a field called title.

YAML

handler:
  class: TitleGeneratorTransformer
  toField: title
  titleMaxLength: 200
  detectHeading: true

JSON

{
  "handler": {
    "class": "TitleGeneratorTransformer",
    "toField": "title",
    "titleMaxLength": 200,
    "detectHeading": true
  }
}

XML

<handler>
  <class>TitleGeneratorTransformer</class>
  <toField>title</toField>
  <titleMaxLength>200</titleMaxLength>
  <detectHeading>true</detectHeading>
</handler>

Usage

Full configuration skeleton, with every option and its default

YAML

class: TitleGeneratorTransformer
detectHeading: false
detectHeadingMaxLength: 0
detectHeadingMinLength: 0
fieldMatcher:
  ignoreCase: false
  ignoreDiacritic: false
  matchEmpty: false
  method: BASIC
  negateMatches: false
  partial: false
  pattern: string
  replaceAll: false
  trim: false
fromField: string
label: string
maxReadSize: 0
onSet: APPEND
sourceCharset: string
titleMaxLength: 0
toField: string

JSON

{
  "class": "TitleGeneratorTransformer",
  "detectHeading": false,
  "detectHeadingMaxLength": 0,
  "detectHeadingMinLength": 0,
  "fieldMatcher": {
    "ignoreCase": false,
    "ignoreDiacritic": false,
    "matchEmpty": false,
    "method": "BASIC",
    "negateMatches": false,
    "partial": false,
    "pattern": "string",
    "replaceAll": false,
    "trim": false
  },
  "fromField": "string",
  "label": "string",
  "maxReadSize": 0,
  "onSet": "APPEND",
  "sourceCharset": "string",
  "titleMaxLength": 0,
  "toField": "string"
}

XML

<handler>
  <class>TitleGeneratorTransformer</class>
  <detectHeading>false</detectHeading>
  <detectHeadingMaxLength>0</detectHeadingMaxLength>
  <detectHeadingMinLength>0</detectHeadingMinLength>
  <fieldMatcher>
    <ignoreCase>false</ignoreCase>
    <ignoreDiacritic>false</ignoreDiacritic>
    <matchEmpty>false</matchEmpty>
    <method>BASIC</method>
    <negateMatches>false</negateMatches>
    <partial>false</partial>
    <pattern>string</pattern>
    <replaceAll>false</replaceAll>
    <trim>false</trim>
  </fieldMatcher>
  <fromField>string</fromField>
  <label>string</label>
  <maxReadSize>0</maxReadSize>
  <onSet>APPEND</onSet>
  <sourceCharset>string</sourceCharset>
  <titleMaxLength>0</titleMaxLength>
  <toField>string</toField>
</handler>

Properties

Property	Type	Required	Default
detectHeading	boolean	No	false
detectHeadingMaxLength	integer	No	150
detectHeadingMinLength	integer	No	10
fieldMatcher	TextMatcher	No	TextMatcher
fromField	string	No	-
label	string	No	-
maxReadSize	integer	No	10000000
onSet	enum	No	-
sourceCharset	string	No	-
titleMaxLength	integer	No	150
toField	string	No	document.generatedTitle

Property Details

detectHeading

Determines if headings in the document should be prioritized for title generation

detectHeadingMaxLength

Specifies the maximum length of a heading to be considered for title generation.

detectHeadingMinLength

Shortest run of text that may be taken as a heading when detectHeading is on. Default is 10 characters, which filters out stray short lines that are rarely titles.

fieldMatcher

Matches document fields based on specified patterns and methods

fromField

Field whose text the title is generated from. Defaults to the document content, which is the usual choice — set this only when the text you want a title for lives in a field.

label

An optional, user-supplied label for this step in the importer pipeline. It has no effect on processing — it exists purely to help identify this step, for example in large configurations or in the visual configurator.

maxReadSize

Max characters to read at once

onSet

What to do if a target field of the same name already exist.

Allowed Values

• APPEND
• PREPEND
• REPLACE
• OPTIONAL

sourceCharset

Sets the character encoding of the source document to ensure proper reading and processing.

titleMaxLength

Sets the maximum allowable length for the generated title.

toField

A target field name used to store the value.