JsonFileCommitter

Local files of JSON format.

committer/coreJavadoc: JsonFileCommitter

Notes

File Format

The generated file content will contain any number of "upsert" and "delete" objects. Both objects will contain the document "reference" and "metadata". The metadata is made of names followed by an undefined number of values as an array. The "upsert" entries also have a "content" property. Like this example containing 1 "upsert" and 1 "delete":

[
  {"upsert": {
    "reference": "https://example.com/my-document.html",
    "metadata": {
      "title": ["My Example Document"],
      "keywords": [
        "Example",
        "Demo"
      ],
    },
    "content": "Document content goes here."
  }},
  {"delete": {
    "reference": "https://example.com/i-was-deleted.html",
    "metadata": {
      "contentType": ["text/html"],
    }
  }},
]

Examples

YAML

class: JsonFileCommitter
filePath: /path/to/committer.json

JSON

{
  "class": "JsonFileCommitter",
  "filePath": "/path/to/committer.json"
}

XML

<class>JsonFileCommitter</class>
<filePath>/path/to/committer.json</filePath>

Usage

Full configuration skeleton, with every option and its default

YAML

class: JsonFileCommitter
compress: false
directory: string
docsPerFile: 0
fieldMappings: {}
fileNamePrefix: string
fileNameSuffix: string
indent: 0
restrictions:
  - fieldMatcher: {}
    valueMatcher: {}
splitUpsertDelete: false

JSON

{
  "class": "JsonFileCommitter",
  "compress": false,
  "directory": "string",
  "docsPerFile": 0,
  "fieldMappings": {},
  "fileNamePrefix": "string",
  "fileNameSuffix": "string",
  "indent": 0,
  "restrictions": [
    {
      "fieldMatcher": {},
      "valueMatcher": {}
    }
  ],
  "splitUpsertDelete": false
}

XML

<jsonFileCommitter>
  <class>JsonFileCommitter</class>
  <compress>false</compress>
  <directory>string</directory>
  <docsPerFile>0</docsPerFile>
  <fieldMappings/>
  <fileNamePrefix>string</fileNamePrefix>
  <fileNameSuffix>string</fileNameSuffix>
  <indent>0</indent>
  <restrictions>
    <restriction>
      <fieldMatcher/>
      <valueMatcher/>
    </restriction>
  </restrictions>
  <splitUpsertDelete>false</splitUpsertDelete>
</jsonFileCommitter>

Properties

Property	Type	Required	Default
compress	boolean	No	false
directory	string	No	-
docsPerFile	integer	No	0
fieldMappings	Record<string, string>	No	-
fileNamePrefix	string	No	-
fileNameSuffix	string	No	-
indent	integer	No	-1
restrictions	PropertyMatcher[]	No	-
splitUpsertDelete	boolean	No	false

Property Details

compress

Compress the generated JSON files using the GZIP file format.

directory

Local path where JSON files will be written.

docsPerFile

Maximum number of documents per JSON files.

fieldMappings

Mapping of source fields to target JSON properties.

fileNamePrefix

A string prepended to generated file names to distinguish them from other output files.

fileNameSuffix

A string appended to generated file names (before the extension) to distinguish them from other output files.

indent

Number of indentation spaces. Default does not indent.

restrictions

Defines filter conditions that restrict which documents this committer processes. Documents not matching the restrictions are ignored by this committer.

splitUpsertDelete

When enabled, a single update operation is split into separate upsert and delete operations for compatibility with the target system.