AzureSearchCommitter

Commit documents to Microsoft Azure Search.

committer/azurecognitivesearchJavadoc: AzureSearchCommitter

Notes

Commits crawled documents to a Microsoft Azure Cognitive Search index. Supports field mappings, Base64 document key encoding, Windows authentication, and configurable handling of response and validation errors.

Examples

The following example uses the minimum required settings to configure Microsoft Azure Search Committer.

YAML

class: AzureSearchCommitter
endpoint: https://example.search.windows.net
apiKey: 1234567890ABCDEF1234567890ABCDEF
indexName: sample-index

JSON

{
  "class": "AzureSearchCommitter",
  "endpoint": "https://example.search.windows.net",
  "apiKey": "1234567890ABCDEF1234567890ABCDEF",
  "indexName": "sample-index"
}

XML

<committer>
  <class>AzureSearchCommitter</class>
  <endpoint>https://example.search.windows.net</endpoint>
  <apiKey>1234567890ABCDEF1234567890ABCDEF</apiKey>
  <indexName>sample-index</indexName>
</committer>

Usage

Full configuration skeleton, with every option and its default

YAML

class: AzureSearchCommitter
apiKey: string
apiVersion: string
arrayFields: string
arrayFieldsRegex: false
disableDocKeyEncoding: false
endpoint: string
fieldMappings: {}
ignoreResponseErrors: false
ignoreValidationErrors: false
indexName: string
proxySettings:
  credentials:
    password: string
    passwordKey: {}
    username: string
  host:
    name: string
    port: 0
  realm: string
  scheme: string
queue:
  class: FsQueue
  batchSize: 0
  commitLeftoversOnInit: false
  maxPerFolder: 0
  onCommitFailure:
    ignoreErrors: false
    maxRetries: 0
    retryDelay: 0
    splitBatch: OFF
restrictions:
  - fieldMatcher: {}
    valueMatcher: {}
sourceKeyField: string
targetContentField: string
targetKeyField: string
useWindowsAuth: false

JSON

{
  "class": "AzureSearchCommitter",
  "apiKey": "string",
  "apiVersion": "string",
  "arrayFields": "string",
  "arrayFieldsRegex": false,
  "disableDocKeyEncoding": false,
  "endpoint": "string",
  "fieldMappings": {},
  "ignoreResponseErrors": false,
  "ignoreValidationErrors": false,
  "indexName": "string",
  "proxySettings": {
    "credentials": {
      "password": "string",
      "passwordKey": {},
      "username": "string"
    },
    "host": {
      "name": "string",
      "port": 0
    },
    "realm": "string",
    "scheme": "string"
  },
  "queue": {
    "class": "FsQueue",
    "batchSize": 0,
    "commitLeftoversOnInit": false,
    "maxPerFolder": 0,
    "onCommitFailure": {
      "ignoreErrors": false,
      "maxRetries": 0,
      "retryDelay": 0,
      "splitBatch": "OFF"
    }
  },
  "restrictions": [
    {
      "fieldMatcher": {},
      "valueMatcher": {}
    }
  ],
  "sourceKeyField": "string",
  "targetContentField": "string",
  "targetKeyField": "string",
  "useWindowsAuth": false
}

XML

<azureSearchCommitter>
  <class>AzureSearchCommitter</class>
  <apiKey>string</apiKey>
  <apiVersion>string</apiVersion>
  <arrayFields>string</arrayFields>
  <arrayFieldsRegex>false</arrayFieldsRegex>
  <disableDocKeyEncoding>false</disableDocKeyEncoding>
  <endpoint>string</endpoint>
  <fieldMappings/>
  <ignoreResponseErrors>false</ignoreResponseErrors>
  <ignoreValidationErrors>false</ignoreValidationErrors>
  <indexName>string</indexName>
  <proxySettings>
    <credentials>
      <password>string</password>
      <passwordKey/>
      <username>string</username>
    </credentials>
    <host>
      <name>string</name>
      <port>0</port>
    </host>
    <realm>string</realm>
    <scheme>string</scheme>
  </proxySettings>
  <queue>
    <class>FsQueue</class>
    <batchSize>0</batchSize>
    <commitLeftoversOnInit>false</commitLeftoversOnInit>
    <maxPerFolder>0</maxPerFolder>
    <onCommitFailure/>
  </queue>
  <restrictions>
    <restriction>
      <fieldMatcher/>
      <valueMatcher/>
    </restriction>
  </restrictions>
  <sourceKeyField>string</sourceKeyField>
  <targetContentField>string</targetContentField>
  <targetKeyField>string</targetKeyField>
  <useWindowsAuth>false</useWindowsAuth>
</azureSearchCommitter>

Properties

Property	Type	Required	Default
apiKey	string	No	-
apiVersion	string	No	2020-06-30
arrayFields	string	No	-
arrayFieldsRegex	boolean	No	false
disableDocKeyEncoding	boolean	No	false
endpoint	string	No	-
fieldMappings	Record<string, string>	No	-
ignoreResponseErrors	boolean	No	false
ignoreValidationErrors	boolean	No	false
indexName	string	No	-
proxySettings	ProxySettings	No	ProxySettings
queue	CommitterQueue	No	FsQueue
restrictions	PropertyMatcher[]	No	-
sourceKeyField	string	No	-
targetContentField	string	No	content
targetKeyField	string	No	id
useWindowsAuth	boolean	No	false

Property Details

apiKey

Azure Search API admin key.

apiVersion

A specific Azure Search API version to use.

arrayFields

Field whose values are forcefully sent as array, even if they are single values. Unless "Use Regular Expression for Array Fields" is checked, expects a comma-separated list of field names.

arrayFieldsRegex

Use regular expression instead of comma-separated values when defining "Array Fields".

disableDocKeyEncoding

Disable Base64 encoding of the document document reference (Azure Search Document Key). Base64 is the preferred approach when your document key can contain special characters.

endpoint

Azure Search endpoint.

fieldMappings

Mapping of source fields to target ones (Azure Search fields).

ignoreResponseErrors

Log Azure Search response errors instead of throwing an exception.

ignoreValidationErrors

Do not perform validation of field names before sending a batch of documents to Azure Search.

indexName

Name of the target Azure Search index where to send documents.

proxySettings

Configures an optional HTTP proxy for outbound connections. Leave unset to connect directly.

queue

Configures the committer queue used to batch documents before sending them to the target system. Adjust queue settings to tune throughput and memory usage.

restrictions

Defines filter conditions that restrict which documents this committer processes. Documents not matching the restrictions are ignored by this committer.

sourceKeyField

Source document field name containing the value that will be stored in Azure Search target document key field. Default is the document reference.

targetContentField

Name of Azure Search document field name to store document content. Default is "content".

targetKeyField

Name of Azure Search document field where to store a document unique key identifier ("Source Key Field"). Default is "id".

useWindowsAuth

Use Windows authentication to connect.