Package com.norconex.committer.elasticsearch

Class ElasticsearchCommitter

java.lang.Object
com.norconex.committer.core.AbstractCommitter<ElasticsearchCommitterConfig>
com.norconex.committer.core.batch.AbstractBatchCommitter<ElasticsearchCommitterConfig>
com.norconex.committer.elasticsearch.ElasticsearchCommitter
All Implemented Interfaces:
BatchConsumer, Committer, Configurable<ElasticsearchCommitterConfig>, AutoCloseable
public class ElasticsearchCommitter extends AbstractBatchCommitter<ElasticsearchCommitterConfig>

Commits documents to Elasticsearch. This committer relies on Elasticsearch REST API.

"_id" field

Elasticsearch expects a field named "_id" that uniquely identifies each documents. You can provide that field yourself in documents you submit. If you do not specify an "_id" field, this committer will create one for you, using the document reference as the identifier value.

"content" field

By default the "body" of a document is read as an input stream and stored in a "content" field. You can change that target field name with ElasticsearchCommitterConfig.setTargetContentField(String). If you set the target content field to null, it will effectively skip storing the content stream.

Dots (.) in field names

Your Elasticsearch installation may consider dots in field names to be representing "objects", which may not always be what you want. If having dots is causing you issues, make sure not to submit fields with dots, or use ElasticsearchCommitterConfig.setDotReplacement(String) to replace dots with a character of your choice (e.g., underscore). If your dot represents a nested object, keep reading.

JSON Objects

It is possible to provide a regular expression that will identify one or more fields containing a JSON object rather than a regular string (ElasticsearchCommitterConfig.setJsonFieldsPattern(String)). For example, this is a useful way to store nested objects. While very flexible, it can be challenging to come up with the JSON structure. You may want to consider custom code. For this to work properly, make sure you define your Elasticsearch field mappings on your index beforehand.

Elasticsearch ID limitations:

As of this writing, Elasticsearch 5 or higher have a 512 bytes limitation on its "_id" field. By default, an error (from Elasticsearch) will result from trying to submit documents with an invalid ID. You can get around this by setting ElasticsearchCommitterConfig.setFixBadIds(boolean) to true. It will truncate references that are too long and append a hash code to it representing the truncated part. This approach is not 100% collision-free (uniqueness), but it should safely cover the vast majority of cases.

Type Name

As of Elasticsearch 7.0, the index type has been deprecated. If you are using Elasticsearch 7.0 or higher, do not configure the typeName. Doing so may cause errors. The typeName is available only for backward compatibility for those using this Committer with older versions of Elasticsearch.

Authentication

Basic authentication is supported for password-protected clusters. Alternatively, API Key authentication can be used by providing the encoded API key value via ElasticsearchCommitterConfig.setApiKey(String). When an API key is set, it takes precedence over basic credentials. The API key value should be the Base64-encoded string as provided by Elasticsearch (i.e., the value sent in the Authorization: ApiKey ... header).

Timeouts

You can specify timeout values for when this committer sends documents to Elasticsearch.

Author:
Pascal Essiembre