AmazonCloudSearchCommitter

Commit documents to Amazon CloudSearch.

committer/amazoncloudsearchJavadoc: AmazonCloudSearchCommitter

Notes

Commits crawled documents to an Amazon CloudSearch index. Supports field mappings, optional ID normalization for CloudSearch compatibility, and AWS credential configuration via explicit keys or environment variables.

Examples

The following example uses the minimum required settings to configure the Amazon CloudSearch Committer. This example relies on environment variables for AWS keys.

YAML

class: CloudSearchCommitter
serviceEndpoint: search-example-xyz.some-region.cloudsearch.amazonaws.com

JSON

{
  "class": "CloudSearchCommitter",
  "serviceEndpoint": "search-example-xyz.some-region.cloudsearch.amazonaws.com"
}

XML

<committer>
  <class>CloudSearchCommitter</class>
  <serviceEndpoint>search-example-xyz.some-region.cloudsearch.amazonaws.com</serviceEndpoint>
</committer>

Usage

Full configuration skeleton, with every option and its default

YAML

class: AmazonCloudSearchCommitter
accessKey: string
fieldMappings: {}
fixBadIds: false
proxySettings:
  credentials:
    password: string
    passwordKey: {}
    username: string
  host:
    name: string
    port: 0
  realm: string
  scheme: string
queue:
  class: FsQueue
  batchSize: 0
  commitLeftoversOnInit: false
  maxPerFolder: 0
  onCommitFailure:
    ignoreErrors: false
    maxRetries: 0
    retryDelay: 0
    splitBatch: OFF
restrictions:
  - fieldMatcher: {}
    valueMatcher: {}
secretKey: string
serviceEndpoint: string
signingRegion: string
sourceIdField: string
targetContentField: string

JSON

{
  "class": "AmazonCloudSearchCommitter",
  "accessKey": "string",
  "fieldMappings": {},
  "fixBadIds": false,
  "proxySettings": {
    "credentials": {
      "password": "string",
      "passwordKey": {},
      "username": "string"
    },
    "host": {
      "name": "string",
      "port": 0
    },
    "realm": "string",
    "scheme": "string"
  },
  "queue": {
    "class": "FsQueue",
    "batchSize": 0,
    "commitLeftoversOnInit": false,
    "maxPerFolder": 0,
    "onCommitFailure": {
      "ignoreErrors": false,
      "maxRetries": 0,
      "retryDelay": 0,
      "splitBatch": "OFF"
    }
  },
  "restrictions": [
    {
      "fieldMatcher": {},
      "valueMatcher": {}
    }
  ],
  "secretKey": "string",
  "serviceEndpoint": "string",
  "signingRegion": "string",
  "sourceIdField": "string",
  "targetContentField": "string"
}

XML

<amazonCloudSearchCommitter>
  <class>AmazonCloudSearchCommitter</class>
  <accessKey>string</accessKey>
  <fieldMappings/>
  <fixBadIds>false</fixBadIds>
  <proxySettings>
    <credentials>
      <password>string</password>
      <passwordKey/>
      <username>string</username>
    </credentials>
    <host>
      <name>string</name>
      <port>0</port>
    </host>
    <realm>string</realm>
    <scheme>string</scheme>
  </proxySettings>
  <queue>
    <class>FsQueue</class>
    <batchSize>0</batchSize>
    <commitLeftoversOnInit>false</commitLeftoversOnInit>
    <maxPerFolder>0</maxPerFolder>
    <onCommitFailure/>
  </queue>
  <restrictions>
    <restriction>
      <fieldMatcher/>
      <valueMatcher/>
    </restriction>
  </restrictions>
  <secretKey>string</secretKey>
  <serviceEndpoint>string</serviceEndpoint>
  <signingRegion>string</signingRegion>
  <sourceIdField>string</sourceIdField>
  <targetContentField>string</targetContentField>
</amazonCloudSearchCommitter>

Properties

Property	Type	Required	Default
accessKey	string	No	-
fieldMappings	Record<string, string>	No	-
fixBadIds	boolean	No	false
proxySettings	ProxySettings	No	ProxySettings
queue	CommitterQueue	No	FsQueue
restrictions	PropertyMatcher[]	No	-
secretKey	string	No	-
serviceEndpoint	string	No	-
signingRegion	string	No	-
sourceIdField	string	No	-
targetContentField	string	No	content

Property Details

accessKey

CloudSearch access key. Leave blank to take it from the environment.

fieldMappings

Mapping of source fields to target ones (index field name).

fixBadIds

Forces references to fit into a CloudSearch id field. Truncate references that are too long and append a hash code to it to keep uniqueness. It will also convert invalid characters to underscore. This approach is not 100% collision-free (uniqueness), but it should safely cover the vast majority of cases.

proxySettings

Configures an optional HTTP proxy for outbound connections. Leave unset to connect directly.

queue

Configures the committer queue used to batch documents before sending them to the target system. Adjust queue settings to tune throughput and memory usage.

restrictions

Defines filter conditions that restrict which documents this committer processes. Documents not matching the restrictions are ignored by this committer.

secretKey

CloudSearch secret key. Leave blank to take it from the environment.

serviceEndpoint

CloudSearch service endpoint.

signingRegion

CloudSearch signing region.

sourceIdField

Document field name containing the value that will be stored in CloudSearch "id" field. Default is the document reference.

targetContentField

CloudSearch field name to store the document content. Default is "content".