ElasticsearchCommitter
Commit documents to Elasticsearch or Amazon OpenSearch.
Notes
Commits crawled documents to an Elasticsearch or Amazon OpenSearch index. Supports cluster node discovery, field mappings, dot replacement in field names, JSON field handling, and optional ID fix-up for documents with long references.
Examples
The following example uses the minimum required settings to configure Elasticsearch (or AWS OpenSearch) Committer.
- YAML
- JSON
- XML
class: ElasticsearchCommitter
indexName: some_index
{
"class": "ElasticsearchCommitter",
"indexName": "some_index"
}
<committer>
<class>ElasticsearchCommitter</class>
<indexName>some_index</indexName>
</committer>
Usage
Full configuration skeleton, with every option and its default
- YAML
- JSON
- XML
class: ElasticsearchCommitter
apiKey: string
connectionTimeout: 0
credentials:
password: string
passwordKey:
size: 0
source: KEY
value: string
username: string
discoverNodes: false
dotReplacement: string
fieldMappings: {}
fixBadIds: false
ignoreResponseErrors: false
indexName: string
jsonFieldsPattern: string
nodes:
- string
queue:
class: FsQueue
batchSize: 0
commitLeftoversOnInit: false
maxPerFolder: 0
onCommitFailure:
ignoreErrors: false
maxRetries: 0
retryDelay: 0
splitBatch: OFF
restrictions:
- fieldMatcher: {}
valueMatcher: {}
socketTimeout: 0
sourceIdField: string
targetContentField: string
typeName: string
{
"class": "ElasticsearchCommitter",
"apiKey": "string",
"connectionTimeout": 0,
"credentials": {
"password": "string",
"passwordKey": {
"size": 0,
"source": "KEY",
"value": "string"
},
"username": "string"
},
"discoverNodes": false,
"dotReplacement": "string",
"fieldMappings": {},
"fixBadIds": false,
"ignoreResponseErrors": false,
"indexName": "string",
"jsonFieldsPattern": "string",
"nodes": [
"string"
],
"queue": {
"class": "FsQueue",
"batchSize": 0,
"commitLeftoversOnInit": false,
"maxPerFolder": 0,
"onCommitFailure": {
"ignoreErrors": false,
"maxRetries": 0,
"retryDelay": 0,
"splitBatch": "OFF"
}
},
"restrictions": [
{
"fieldMatcher": {},
"valueMatcher": {}
}
],
"socketTimeout": 0,
"sourceIdField": "string",
"targetContentField": "string",
"typeName": "string"
}
<elasticsearchCommitter>
<class>ElasticsearchCommitter</class>
<apiKey>string</apiKey>
<connectionTimeout>0</connectionTimeout>
<credentials>
<password>string</password>
<passwordKey>
<size>0</size>
<source>KEY</source>
<value>string</value>
</passwordKey>
<username>string</username>
</credentials>
<discoverNodes>false</discoverNodes>
<dotReplacement>string</dotReplacement>
<fieldMappings/>
<fixBadIds>false</fixBadIds>
<ignoreResponseErrors>false</ignoreResponseErrors>
<indexName>string</indexName>
<jsonFieldsPattern>string</jsonFieldsPattern>
<nodes>
<node>string</node>
</nodes>
<queue>
<class>FsQueue</class>
<batchSize>0</batchSize>
<commitLeftoversOnInit>false</commitLeftoversOnInit>
<maxPerFolder>0</maxPerFolder>
<onCommitFailure/>
</queue>
<restrictions>
<restriction>
<fieldMatcher/>
<valueMatcher/>
</restriction>
</restrictions>
<socketTimeout>0</socketTimeout>
<sourceIdField>string</sourceIdField>
<targetContentField>string</targetContentField>
<typeName>string</typeName>
</elasticsearchCommitter>
Properties
| Property | Type | Required | Default |
|---|---|---|---|
apiKey | string | No | - |
connectionTimeout | integer | No | 1000 |
credentials | Credentials | No | Credentials |
discoverNodes | boolean | No | false |
dotReplacement | string | No | - |
fieldMappings | Record<string, string> | No | - |
fixBadIds | boolean | No | false |
ignoreResponseErrors | boolean | No | false |
indexName | string | No | - |
jsonFieldsPattern | string | No | - |
nodes | string[] | No | [http://localhost:9200] |
queue | CommitterQueue | No | FsQueue |
restrictions | PropertyMatcher[] | No | - |
socketTimeout | integer | No | 30000 |
sourceIdField | string | No | - |
targetContentField | string | No | content |
typeName | string | No | - |
Property Details
apiKey
API key for authentication with Elasticsearch or Amazon OpenSearch.
connectionTimeout
Maximum wait time for a requested connection to be fully established, in milliseconds. Default is 1 second (1000 ms).
credentials
Username and password to a protected Elasticsearch.
discoverNodes
Enable automatic discovery of Elasticsearch cluster nodes.
dotReplacement
The character used to replace dots in field names. Leave blank to keep the dots in field names (nested fields).
fieldMappings
Mapping of source fields to target ones (Elasticsearch fields).
fixBadIds
Fix IDs that are too long for Elasticsearch ID limitation (512 bytes max). When enabled, long IDs will be truncated and a hash code representing the truncated part will be appended.
ignoreResponseErrors
Log Elasticsearch response errors instead of throwing an exception.
indexName
Name of target Elasticsearch index where documents will be stored.
jsonFieldsPattern
Regular expression matching fields whose values are raw JSON rather than regular strings, such as a JSON object or, for a multi-valued field, a list of numbers like an embedding vector. Values of matching fields are sent as they are, unquoted.
nodes
One or more Elasticsearch nodes to connect to. Defaults to http://localhost:9200.
queue
Configures the committer queue used to batch documents before sending them to the target system. Adjust queue settings to tune throughput and memory usage.
restrictions
Defines filter conditions that restrict which documents this committer processes. Documents not matching the restrictions are ignored by this committer.
socketTimeout
The maximum time in milliseconds to wait for data on an established connection before giving up.
sourceIdField
Source document field name containing the value to be stored in Elasticsearch "_id" field. Leave blank to use the document reference instead of a field (default).
targetContentField
Target Elasticsearch field name where content will be stored. Default is "content". Leave blank to disable storing the content.
typeName
Type name used in older versions of Elasticsearch. Leave blank if using Elasticsearch 7.0. or higher.