ImporterConfig

Configures the document importing pipeline with handlers for parsing, transforming, and filtering documents.

importerJavadoc: ImporterConfig

Notes

Configures the document importing pipeline. Handlers are executed in order and can include transformers, parsers, splitters, and flow control wrappers. Handlers applied before a parser run as pre-parse handlers; those after run as post-parse handlers. Including a parser transitions the pipeline from pre-parse to post-parse mode.

Examples

handlers is a list, and every entry in it is an object with a single key: handler for an ordinary handler, or if / ifNot for a branch. then and else hold lists of those same entries. See Importer Flow Control for the full picture.

The following pipeline parses each document, drops the ones that came out without a title, and tags the rest according to how recent they are.

YAML

handlers:
  - handler:
      class: DefaultParser
  - if:
      condition:
        class: BlankCondition
        fieldMatcher:
          pattern: title
      then:
        - handler:
            class: Reject
            message: No title after parsing
  - if:
      condition:
        class: DateCondition
        fieldMatcher:
          pattern: date_created
        valueMatcher:
          operator: ge
          date: NOW-1Y
      then:
        - handler:
            class: ConstantTransformer
            constants:
              - name: freshness
                values:
                  - recent
      else:
        - handler:
            class: ConstantTransformer
            constants:
              - name: freshness
                values:
                  - archive

JSON

{
  "handlers": [
    {
      "handler": {
        "class": "DefaultParser"
      }
    },
    {
      "if": {
        "condition": {
          "class": "BlankCondition",
          "fieldMatcher": {
            "pattern": "title"
          }
        },
        "then": [
          {
            "handler": {
              "class": "Reject",
              "message": "No title after parsing"
            }
          }
        ]
      }
    },
    {
      "if": {
        "condition": {
          "class": "DateCondition",
          "fieldMatcher": {
            "pattern": "date_created"
          },
          "valueMatcher": {
            "operator": "ge",
            "date": "NOW-1Y"
          }
        },
        "then": [
          {
            "handler": {
              "class": "ConstantTransformer",
              "constants": [
                {
                  "name": "freshness",
                  "values": [
                    "recent"
                  ]
                }
              ]
            }
          }
        ],
        "else": [
          {
            "handler": {
              "class": "ConstantTransformer",
              "constants": [
                {
                  "name": "freshness",
                  "values": [
                    "archive"
                  ]
                }
              ]
            }
          }
        ]
      }
    }
  ]
}

XML

<handlers>
  <handler>
    <class>DefaultParser</class>
  </handler>
  <if>
    <condition>
      <class>BlankCondition</class>
      <fieldMatcher>
        <pattern>title</pattern>
      </fieldMatcher>
    </condition>
    <then>
      <handler>
        <class>Reject</class>
        <message>No title after parsing</message>
      </handler>
    </then>
  </if>
  <if>
    <condition>
      <class>DateCondition</class>
      <fieldMatcher>
        <pattern>date_created</pattern>
      </fieldMatcher>
      <valueMatcher>
        <operator>ge</operator>
        <date>NOW-1Y</date>
      </valueMatcher>
    </condition>
    <then>
      <handler>
        <class>ConstantTransformer</class>
        <constants>
          <name>freshness</name>
          <values>recent</values>
        </constants>
      </handler>
    </then>
    <else>
      <handler>
        <class>ConstantTransformer</class>
        <constants>
          <name>freshness</name>
          <values>archive</values>
        </constants>
      </handler>
    </else>
  </if>
</handlers>

Memory limits and the temporary directory only apply when the Importer runs on its own — a crawl supplies its own values.

YAML

maxMemoryInstance: 200000000
maxMemoryPool: 2000000000
tempDir: /var/tmp/importer
handlers:
  - handler:
      class: DefaultParser

JSON

{
  "maxMemoryInstance": 200000000,
  "maxMemoryPool": 2000000000,
  "tempDir": "/var/tmp/importer",
  "handlers": [
    {
      "handler": {
        "class": "DefaultParser"
      }
    }
  ]
}

XML

<maxMemoryInstance>200000000</maxMemoryInstance>
<maxMemoryPool>2000000000</maxMemoryPool>
<tempDir>/var/tmp/importer</tempDir>
<handlers>
  <handler>
    <class>DefaultParser</class>
  </handler>
</handlers>

Usage

Full configuration skeleton, with every option and its default

YAML

handlers: []
maxMemoryInstance: 0
maxMemoryPool: 0
responseProcessors:
  - noop: string
tempDir: string

JSON

{
  "handlers": [],
  "maxMemoryInstance": 0,
  "maxMemoryPool": 0,
  "responseProcessors": [
    {
      "noop": "string"
    }
  ],
  "tempDir": "string"
}

XML

<importer>
  <handlers>
    <handler/>
  </handlers>
  <maxMemoryInstance>0</maxMemoryInstance>
  <maxMemoryPool>0</maxMemoryPool>
  <responseProcessors>
    <responseProcessor>
      <noop>string</noop>
    </responseProcessor>
  </responseProcessors>
  <tempDir>string</tempDir>
</importer>

Properties

Property	Type	Required	Default
handlers	DocHandler[]	No	[DefaultParser]
maxMemoryInstance	integer	No	100000000
maxMemoryPool	integer	No	1000000000
responseProcessors	ImporterResponseProcessor[]	No	-
tempDir	string	No	-

Property Details

handlers

Specify handlers for imported documents in their original format or applied before or after parsing. Combine with condition wrappers to create a processing "flowControl"

maxMemoryInstance

Maximum bytes a single document may cache in memory before it spills to tempDir. Default is 100000000 (100 MB). Raising it trades memory for fewer disk writes on large documents; lowering it protects a small heap when documents are big.

Only applies when the Importer runs on its own — from the command line, or embedded via Importer#importDocument(ImporterRequest). Documents arriving from a crawl already carry their own cache settings.

maxMemoryPool

Maximum bytes shared by every document being processed by Importer instances in the same JVM. Default is 1000000000 (1 GB). This is the ceiling maxMemoryInstance allocations draw from.

Same scope as maxMemoryInstance: standalone or embedded Importer use only.

responseProcessors

One or more optional custom classes that processes an Importer response to modify it or perform other actions as required before it is returned.

tempDir

Where documents too large for the memory cache are written while they are processed. Defaults to the system temporary directory. Point it at a fast local disk with room for your largest documents.

Same scope as the memory settings: standalone or embedded Importer use only — a crawl supplies its own working directory.