ScriptTransformer

Transformer incoming documents using a scripting language.

importerJavadoc: ScriptTransformer

Notes

The default script engine is JavaScript.

Refer to ScriptRunner for more information on using a scripting language with Norconex Importer.

How to tag documents with scripting:

The following are variables made available to your script for each document:

• reference: Document unique reference as a string.
• content: Document content, as a string (of maxReadSize length).
• metadata: Document metadata as an Properties object.
• parsed: Whether the document was already parsed, as a boolean.
• sectionIndex: Content section index if it had to be split, as an integer. There are no expected return value from your script. Returning one has no effect.

Examples

The following examples add new metadata field indicating which fruit is the document about.

JavaScript

YAML

handler:
  class: ScriptTransformer
  script: <![CDATA[metadata.add('fruit', 'apple');]]

JSON

{
  "handler": {
    "class": "ScriptTransformer",
    "script": "<![CDATA[metadata.add('fruit', 'apple');]]"
  }
}

XML

<handler>
  <class>ScriptTransformer</class>
  <script><![CDATA[metadata.add('fruit', 'apple');]]</script>
</handler>

lua

LUA

YAML

handler:
  class: ScriptTransformer
  engineName: LUA
  script: <![CDATA[metadata:addString('fruit', {'apple'});]]>

JSON

{
  "handler": {
    "class": "ScriptTransformer",
    "engineName": "LUA",
    "script": "<![CDATA[metadata:addString('fruit', {'apple'});]]>"
  }
}

XML

<handler>
  <class>ScriptTransformer</class>
  <engineName>LUA</engineName>
  <script><![CDATA[metadata:addString('fruit', {'apple'});]]></script>
</handler>

Usage

Full configuration skeleton, with every option and its default

YAML

class: ScriptTransformer
engineName: string
fieldMatcher:
  ignoreCase: false
  ignoreDiacritic: false
  matchEmpty: false
  method: BASIC
  negateMatches: false
  partial: false
  pattern: string
  replaceAll: false
  trim: false
label: string
maxReadSize: 0
script: string
sourceCharset: string

JSON

{
  "class": "ScriptTransformer",
  "engineName": "string",
  "fieldMatcher": {
    "ignoreCase": false,
    "ignoreDiacritic": false,
    "matchEmpty": false,
    "method": "BASIC",
    "negateMatches": false,
    "partial": false,
    "pattern": "string",
    "replaceAll": false,
    "trim": false
  },
  "label": "string",
  "maxReadSize": 0,
  "script": "string",
  "sourceCharset": "string"
}

XML

<handler>
  <class>ScriptTransformer</class>
  <engineName>string</engineName>
  <fieldMatcher>
    <ignoreCase>false</ignoreCase>
    <ignoreDiacritic>false</ignoreDiacritic>
    <matchEmpty>false</matchEmpty>
    <method>BASIC</method>
    <negateMatches>false</negateMatches>
    <partial>false</partial>
    <pattern>string</pattern>
    <replaceAll>false</replaceAll>
    <trim>false</trim>
  </fieldMatcher>
  <label>string</label>
  <maxReadSize>0</maxReadSize>
  <script>string</script>
  <sourceCharset>string</sourceCharset>
</handler>

Properties

Property	Type	Required	Default
engineName	string	No	-
fieldMatcher	TextMatcher	No	TextMatcher
label	string	No	-
maxReadSize	integer	No	10000000
script	string	No	-
sourceCharset	string	No	-

Property Details

engineName

Which scripting engine runs the script. JavaScript, lua and velocity are available out of the box; any other JSR-223 engine on the classpath can be named here, such as python with Jython installed. Required — there is no default.

fieldMatcher

Matches document fields based on specified patterns and methods

label

An optional, user-supplied label for this step in the importer pipeline. It has no effect on processing — it exists purely to help identify this step, for example in large configurations or in the visual configurator.

maxReadSize

Max characters to read at once

script

The script content or path to execute as part of this handler's processing logic.

sourceCharset

Sets the character encoding of the source document to ensure proper reading and processing.