CsvFileCommitter

Local comma-separated values files.

committer/coreJavadoc: CsvFileCommitter

Notes

Commits crawled documents to local comma-separated values (CSV) files. Supports configurable delimiters, quoting, column mappings, multi-value joining, file compression, and output directory settings.

Examples

YAML

class: CsvFileCommitter
filePath: /path/to/committer.csv

JSON

{
  "class": "CsvFileCommitter",
  "filePath": "/path/to/committer.csv"
}

XML

<class>CsvFileCommitter</class>
<filePath>/path/to/committer.csv</filePath>

Usage

Full configuration skeleton, with every option and its default

YAML

class: CsvFileCommitter
columns:
  - field: string
    header: string
    truncateAt: 0
compress: false
delimiter: string
directory: string
docsPerFile: 0
escape: string
fieldMappings: {}
fileNamePrefix: string
fileNameSuffix: string
format: DEFAULT
multiValueJoinDelimiter: string
quote: string
restrictions:
  - fieldMatcher: {}
    valueMatcher: {}
showHeaders: false
splitUpsertDelete: false
truncateAt: 0
typeHeader: string

JSON

{
  "class": "CsvFileCommitter",
  "columns": [
    {
      "field": "string",
      "header": "string",
      "truncateAt": 0
    }
  ],
  "compress": false,
  "delimiter": "string",
  "directory": "string",
  "docsPerFile": 0,
  "escape": "string",
  "fieldMappings": {},
  "fileNamePrefix": "string",
  "fileNameSuffix": "string",
  "format": "DEFAULT",
  "multiValueJoinDelimiter": "string",
  "quote": "string",
  "restrictions": [
    {
      "fieldMatcher": {},
      "valueMatcher": {}
    }
  ],
  "showHeaders": false,
  "splitUpsertDelete": false,
  "truncateAt": 0,
  "typeHeader": "string"
}

XML

<csvFileCommitter>
  <class>CsvFileCommitter</class>
  <columns>
    <column>
      <field>string</field>
      <header>string</header>
      <truncateAt>0</truncateAt>
    </column>
  </columns>
  <compress>false</compress>
  <delimiter>string</delimiter>
  <directory>string</directory>
  <docsPerFile>0</docsPerFile>
  <escape>string</escape>
  <fieldMappings/>
  <fileNamePrefix>string</fileNamePrefix>
  <fileNameSuffix>string</fileNameSuffix>
  <format>DEFAULT</format>
  <multiValueJoinDelimiter>string</multiValueJoinDelimiter>
  <quote>string</quote>
  <restrictions>
    <restriction>
      <fieldMatcher/>
      <valueMatcher/>
    </restriction>
  </restrictions>
  <showHeaders>false</showHeaders>
  <splitUpsertDelete>false</splitUpsertDelete>
  <truncateAt>0</truncateAt>
  <typeHeader>string</typeHeader>
</csvFileCommitter>

Properties

Property	Type	Required	Default
columns	CsvColumn[]	No	-
compress	boolean	No	false
delimiter	string	No	-
directory	string	No	-
docsPerFile	integer	No	0
escape	string	No	-
fieldMappings	Record<string, string>	No	-
fileNamePrefix	string	No	-
fileNameSuffix	string	No	-
format	enum	No	-
multiValueJoinDelimiter	string	No	-
quote	string	No	-
restrictions	PropertyMatcher[]	No	-
showHeaders	boolean	No	false
splitUpsertDelete	boolean	No	false
truncateAt	integer	No	5096
typeHeader	string	No	-

Property Details

columns

Target columns for desired document field values.

compress

Compress the generated CSV files using the GZIP file format.

delimiter

Delimiter

directory

Local path where CSV files will be written.

docsPerFile

Maximum number of documents per CSV files.

escape

The escape character to use when needed.

fieldMappings

Mapping of source fields to target ones (CSV columns).

fileNamePrefix

A string prepended to generated file names to distinguish them from other output files.

fileNameSuffix

A string appended to generated file names (before the extension) to distinguish them from other output files.

format

Configuration presets optimized for specific applications consuming CSV files.

Allowed Values

• DEFAULT
• EXCEL
• INFORMIX_UNLOAD1_3
• INFORMIX_UNLOAD_CSV1_3
• MONGO_CSV1_7
• MONGO_TSV1_7
• MYSQL
• ORACLE1_6
• POSTGRESSQL_CSV1_5
• POSTGRESSQL_TEXT1_5
• RFC_4180
• TDF

multiValueJoinDelimiter

One or more characters to be used as a delimiter when combining multi-value fields to become single-value.

quote

A character used to quote values.

restrictions

Defines filter conditions that restrict which documents this committer processes. Documents not matching the restrictions are ignored by this committer.

showHeaders

Write column headers at the first of a generated document.

splitUpsertDelete

When enabled, a single update operation is split into separate upsert and delete operations for compatibility with the target system.

truncateAt

Truncate values larger than N characters. Default is 5096. Use -1 for unlimited.

typeHeader

Name of the column holding the operation type (upsert or delete). Default is type.