ExternalParser

Parses and extracts text from a file using an external application

importerJavadoc: ExternalParser

Notes

Content and metadata are extracted from documents using an external parser. The ExternalParser functions similarly to the ExternalTransformer, but with a key difference: after parsing, the document is marked with the status "parsed". As a result, only handlers that support post-parse processing can be applied afterward. In contrast, the ExternalTransformer does not alter the document’s status. To use an external application to change a file content after parsing has already occurred, consider using ExternalTransformer instead.

Built-in baseline, not a hard limit

ExternalParser is part of the built-in importer toolbox, and can be combined with custom external applications and extraction logic for customer-specific needs.

Command-line arguments:

When constructing the command to launch the external application, it will look for specific tokens to be replaced by file paths arguments (in addition to other arguments you may have). The path arguments are created by this class. They are case-sensitive and the file they represent are temporary (will be deleted after they have been dealt with). It is possible to omit one or more tokens to use standard streams instead where applicable.

Tokens supported by:

${INPUT}

Path to document to be handled by the external application. When omitted, the document content is sent to the external application using the standard input stream (STDIN).

${INPUT_META}

Path to file containing metadata information available so far for the document to be handled by the external application. By default in JSON format. When omitted, no metadata will be made available to the external application.

${OUTPUT}

Path to document resulting from this external handler. When omitted, the output content will be read from the external application standard output (STDOUT).

${OUTPUT_META}

Path to file containing new metadata for the document. By default, the expected format is JSON. When omitted, any metadata extraction patterns defined will be applied against both the external program standard output (STDOUT) and standard error (STDERR). If no patterns are defined, it is assumed no new metadata resulted from the external application.

${REFERENCE}

Unique reference to the document being handled (URL, original file system location, etc.). When omitted, the document reference will not be made available to the external application.

Examples

The following example invokes an external application processing for simple text files that accepts two files as arguments: the first one being the file to transform, the second one being holding the transformation result. It also extract a document number from STDOUT, found as "DocNo:1234" and storing it as "docnumber". The fieldGroup and valueGroup values reference capture group indices in the regex pattern — here group 1 captures the field name and group 2 captures the value.

YAML

handler:
  class: ExternalParser
  command: /path/transform/app ${INPUT} ${OUTPUT}
  environmentVariables:
    ENV: dev
    VERSION: 2.0.3
  extractionPatterns:
    - fieldGroup: 1
      regex:
        pattern: (DocNo):(\d+)
      toField: docnumber
      valueGroup: 2

JSON

{
  "handler": {
    "class": "ExternalParser",
    "command": "/path/transform/app ${INPUT} ${OUTPUT}",
    "environmentVariables": {
      "ENV": "dev",
      "VERSION": "2.0.3"
    },
    "extractionPatterns": [
      {
        "fieldGroup": 1,
        "regex": {
          "pattern": "(DocNo):(\\d+)"
        },
        "toField": "docnumber",
        "valueGroup": 2
      }
    ]
  }
}

XML

<handler>
  <class>ExternalParser</class>
  <command>/path/transform/app ${INPUT} ${OUTPUT}</command>
  <environmentVariables>
    <ENV>dev</ENV>
    <VERSION>2.0.3</VERSION>
  </environmentVariables>
  <extractionPatterns>
    <extractionPattern>
      <fieldGroup>1</fieldGroup>
      <regex>
        <pattern>(DocNo):(d+)</pattern>
      </regex>
      <toField>docnumber</toField>
      <valueGroup>2</valueGroup>
    </extractionPattern>
  </extractionPatterns>
</handler>

Usage

Full configuration skeleton, with every option and its default

YAML

class: ExternalParser
command: string
environmentVariables: {}
extractionPatterns:
  - fieldGroup: 0
    onSet: APPEND
    regex: {}
    toField: string
    valueGroup: 0
label: string
metadataInputFormat: string
metadataOutputFormat: string
onSet: APPEND
tempDir: string

JSON

{
  "class": "ExternalParser",
  "command": "string",
  "environmentVariables": {},
  "extractionPatterns": [
    {
      "fieldGroup": 0,
      "onSet": "APPEND",
      "regex": {},
      "toField": "string",
      "valueGroup": 0
    }
  ],
  "label": "string",
  "metadataInputFormat": "string",
  "metadataOutputFormat": "string",
  "onSet": "APPEND",
  "tempDir": "string"
}

XML

<handler>
  <class>ExternalParser</class>
  <command>string</command>
  <environmentVariables/>
  <extractionPatterns>
    <extractionPattern>
      <fieldGroup>0</fieldGroup>
      <onSet>APPEND</onSet>
      <regex/>
      <toField>string</toField>
      <valueGroup>0</valueGroup>
    </extractionPattern>
  </extractionPatterns>
  <label>string</label>
  <metadataInputFormat>string</metadataInputFormat>
  <metadataOutputFormat>string</metadataOutputFormat>
  <onSet>APPEND</onSet>
  <tempDir>string</tempDir>
</handler>

Properties

Property	Type	Required	Default
command	string	No	-
environmentVariables	Record<string, string>	No	-
extractionPatterns	RegexFieldValueExtractor[]	No	-
label	string	No	-
metadataInputFormat	string	No	json
metadataOutputFormat	string	No	json
onSet	enum	No	-
tempDir	string	No	-

Property Details

command

The command line to execute, with ${INPUT} and ${OUTPUT} placeholders for the document going in and coming out. Escape spaces in the executable path and its arguments.

Sample command line

c:\Apps\myapp.exe ${INPUT} ${OUTPUT} ${INPUT_META} ${OUTPUT_META} ${REFERENCE}

environmentVariables

Defines environment variables to set when executing the external command.

extractionPatterns

Pattern only used when metadataOutputFormat is not specified

label

An optional, user-supplied label for this step in the importer pipeline. It has no effect on processing — it exists purely to help identify this step, for example in large configurations or in the visual configurator.

metadataInputFormat

Sets the format of the metadata input file sent to the external application. One of json (default), xml, or properties is expected. Only applicable when the ${INPUT} token is part of the command.

metadataOutputFormat

Sets the format of the metadata output file from the external application. One of json (default), xml, or properties is expected. Set to null for relying metadata extraction patterns instead. Only applicable when the ${OUTPUT} token is part of the command.

onSet

What to do if a target field of the same name already exist.

Allowed Values

• APPEND
• PREPEND
• REPLACE
• OPTIONAL

tempDir

Sets the directory where to store temporary files used for the parser.