SolrCommitter

Commit documents to Apache Solr search engine.

committer/solrJavadoc: SolrCommitter

Notes

Commits crawled documents to an Apache Solr search engine. Supports multiple Solr client types, field mappings, optional explicit Solr commit control, custom URL parameters, and credential configuration.

Examples

YAML

class: SolrCommitter
solrUrl: http://localhost:8983/solr
solrCollection: mycollection

JSON

{
  "class": "SolrCommitter",
  "solrUrl": "http://localhost:8983/solr",
  "solrCollection": "mycollection"
}

XML

<class>SolrCommitter</class>
<solrUrl>http://localhost:8983/solr</solrUrl>
<solrCollection>mycollection</solrCollection>

Usage

Full configuration skeleton, with every option and its default

YAML

class: SolrCommitter
credentials:
  password: string
  passwordKey:
    size: 0
    source: KEY
    value: string
  username: string
fieldMappings: {}
queue:
  class: FsQueue
  batchSize: 0
  commitLeftoversOnInit: false
  maxPerFolder: 0
  onCommitFailure:
    ignoreErrors: false
    maxRetries: 0
    retryDelay: 0
    splitBatch: OFF
restrictions:
  - fieldMatcher: {}
    valueMatcher: {}
solrClientType: CloudSolrClient
solrCommitDisabled: false
solrURL: string
sourceIdField: string
targetContentField: string
targetIdField: string
updateUrlParams: {}

JSON

{
  "class": "SolrCommitter",
  "credentials": {
    "password": "string",
    "passwordKey": {
      "size": 0,
      "source": "KEY",
      "value": "string"
    },
    "username": "string"
  },
  "fieldMappings": {},
  "queue": {
    "class": "FsQueue",
    "batchSize": 0,
    "commitLeftoversOnInit": false,
    "maxPerFolder": 0,
    "onCommitFailure": {
      "ignoreErrors": false,
      "maxRetries": 0,
      "retryDelay": 0,
      "splitBatch": "OFF"
    }
  },
  "restrictions": [
    {
      "fieldMatcher": {},
      "valueMatcher": {}
    }
  ],
  "solrClientType": "CloudSolrClient",
  "solrCommitDisabled": false,
  "solrURL": "string",
  "sourceIdField": "string",
  "targetContentField": "string",
  "targetIdField": "string",
  "updateUrlParams": {}
}

XML

<solrCommitter>
  <class>SolrCommitter</class>
  <credentials>
    <password>string</password>
    <passwordKey>
      <size>0</size>
      <source>KEY</source>
      <value>string</value>
    </passwordKey>
    <username>string</username>
  </credentials>
  <fieldMappings/>
  <queue>
    <class>FsQueue</class>
    <batchSize>0</batchSize>
    <commitLeftoversOnInit>false</commitLeftoversOnInit>
    <maxPerFolder>0</maxPerFolder>
    <onCommitFailure/>
  </queue>
  <restrictions>
    <restriction>
      <fieldMatcher/>
      <valueMatcher/>
    </restriction>
  </restrictions>
  <solrClientType>CloudSolrClient</solrClientType>
  <solrCommitDisabled>false</solrCommitDisabled>
  <solrURL>string</solrURL>
  <sourceIdField>string</sourceIdField>
  <targetContentField>string</targetContentField>
  <targetIdField>string</targetIdField>
  <updateUrlParams/>
</solrCommitter>

Properties

Property	Type	Required	Default
credentials	Credentials	No	Credentials
fieldMappings	Record<string, string>	No	-
queue	CommitterQueue	No	FsQueue
restrictions	PropertyMatcher[]	No	-
solrClientType	enum	No	-
solrCommitDisabled	boolean	No	false
solrURL	string	No	-
sourceIdField	string	No	-
targetContentField	string	No	content
targetIdField	string	No	id
updateUrlParams	Record<string, string>	No	-

Property Details

credentials

Username and password to a protected Solr.

fieldMappings

Mapping of source fields to target ones (Solr fields).

queue

Configures the committer queue used to batch documents before sending them to the target system. Adjust queue settings to tune throughput and memory usage.

restrictions

Defines filter conditions that restrict which documents this committer processes. Documents not matching the restrictions are ignored by this committer.

solrClientType

Type of Solr Client to use. Default client type is Http2SolrClient.

Allowed Values

• CloudSolrClient
• HttpJettySolrClient
• LBJettySolrClient
• ConcurrentUpdateJettySolrClient
• HttpJdkSolrClient
• HttpSolrClient
• LBHttpSolrClient
• ConcurrentUpdateSolrClient

solrCommitDisabled

Disable sending an explicit Solr commit request at the end of every batch sent, relying Solr to perform auto-commit instead.

solrURL

URL to Solr. Influenced by the client type chosen.

sourceIdField

Document field name containing the value that will be stored in Solr target ID field. Default is the document reference.

targetContentField

Solr field name to store a document content. Default is "content".

targetIdField

Solr field name where to store a document unique identifier (source ID field). Default is "id".

updateUrlParams

Parameters to be added on Solr HTTP calls.