Skip to main content

GoogleCloudSearchCommitter

Commit documents to Google Cloud Search.

committer/googlecloudsearchJavadoc: GoogleCloudSearchCommitter

Notes​

Commits crawled documents to a Google Cloud Search data source using the Cloud Search Indexing API, authenticating with a Google service account key. Documents can be uploaded as extracted plain text (default) or as their original raw content, and metadata fields can optionally be mapped to Google Cloud Search ACL principals with support for ACL inheritance from a parent item.

Examples​

The following example uses the minimum required settings to configure the Google Cloud Search Committer.

class: GoogleCloudSearchCommitter
secretKeyPath: /path/to/google-service-account.json
dataSourceId: your-datasource-id

Usage​

Full configuration skeleton, with every option and its default
class: GoogleCloudSearchCommitter
aclInheritance:
  aclInheritanceType: NOT_APPLICABLE
  fromField: string
aclMappings:
  - fromField: string
    principalType: USER
    target: READERS
apiEndpoint: string
applicationName: string
connectorName: string
dataSourceId: string
failOnDeleteNotFound: false
fieldMappings: {}
httpBackoffInitialIntervalMillis: 0
httpBackoffMaxElapsedTimeMillis: 0
httpBackoffMaxIntervalMillis: 0
httpConnectTimeoutMillis: 0
httpMaxRetries: 0
httpReadTimeoutMillis: 0
keepSourceIdField: false
metadataMappings:
  - defaultValue: string
    fromField: string
    keepFromField: false
    toField: title
queue:
  class: FsQueue
  batchSize: 0
  commitLeftoversOnInit: false
  maxPerFolder: 0
  onCommitFailure:
    ignoreErrors: false
    maxRetries: 0
    retryDelay: 0
    splitBatch: OFF
requestMode: SYNCHRONOUS
restrictions:
  - fieldMatcher: {}
    valueMatcher: {}
secretKeyPath: string
sourceIdField: string
structuredDataMappings:
  - field: string
    type: TEXT
uploadFormat: RAW

Properties​

PropertyTypeRequiredDefault
aclInheritanceAclInheritanceMappingNoAclInheritanceMapping
aclMappingsAclMapping[]No-
apiEndpointstringNo-
applicationNamestringNoNorconex Google Cloud Search Committer
connectorNamestringNo-
dataSourceIdstringNo-
failOnDeleteNotFoundbooleanNofalse
fieldMappingsRecord<string, string>No-
httpBackoffInitialIntervalMillisintegerNo-1
httpBackoffMaxElapsedTimeMillisintegerNo-1
httpBackoffMaxIntervalMillisintegerNo-1
httpConnectTimeoutMillisintegerNo-1
httpMaxRetriesintegerNo-1
httpReadTimeoutMillisintegerNo-1
keepSourceIdFieldbooleanNofalse
metadataMappingsMetadataMapping[]No-
queueCommitterQueueNoFsQueue
requestModeenumNoASYNCHRONOUS
restrictionsPropertyMatcher[]No-
secretKeyPathstringNo-
sourceIdFieldstringNo-
structuredDataMappingsStructuredDataMapping[]No-
uploadFormatenumNoRAW

Property Details​

aclInheritance​

Declares which metadata field holds a parent item reference to use for Google Cloud Search ACL inheritance, along with the inheritance type to apply.

aclMappings​

Mappings of metadata fields to Google Cloud Search ACL principals (readers, denied readers, or owners), each specifying the principal type (user, group, or customer).

apiEndpoint​

Overrides the Google Cloud Search API root URL. Mainly useful for testing against a local mock server.

applicationName​

The application name sent to Google Cloud Search. Default is "Norconex Google Cloud Search Committer".

connectorName​

The connector name sent with indexing requests. Defaults to the application name when not set.

dataSourceId​

The Google Cloud Search data source ID to index documents into.

failOnDeleteNotFound​

Whether a delete request targeting an item that does not exist in the index is treated as a failure. Crawlers routinely delete references that were never indexed (rejected, orphan or unmodified documents), and Google Cloud Search answers those with a 404 NOT_FOUND. Since the desired outcome (the item is absent) already holds, such responses are ignored by default. Set to true to have them fail the batch instead.

Only delete requests are affected. A 404 on an index request always fails, since it signals a bad data source or connector name.

fieldMappings​

Mapping of source fields to target ones (Google Cloud Search structured data field names).

httpBackoffInitialIntervalMillis​

Optional exponential backoff initial interval, in milliseconds, applied to retried Google API requests. Use -1 (the default) to keep the Google API client's default.

httpBackoffMaxElapsedTimeMillis​

Optional exponential backoff maximum elapsed time, in milliseconds, applied to retried Google API requests. Use -1 (the default) to keep the Google API client's default.

httpBackoffMaxIntervalMillis​

Optional exponential backoff maximum interval, in milliseconds, applied to retried Google API requests. Use -1 (the default) to keep the Google API client's default.

httpConnectTimeoutMillis​

Optional HTTP connect timeout, in milliseconds, applied to Google API requests. Use -1 (the default) to keep the Google API client's default.

httpMaxRetries​

Optional maximum number of HTTP retries for Google API requests. Use -1 (the default) to keep the Google API client's default.

httpReadTimeoutMillis​

Optional HTTP read timeout, in milliseconds, applied to Google API requests. Use -1 (the default) to keep the Google API client's default.

keepSourceIdField​

Whether to keep the source ID field in the document metadata after it has been extracted.

metadataMappings​

Mappings used to populate Google Cloud Search predefined ItemMetadata entries. Each mapping supports:

  • fromField (optional): crawler metadata source field.
  • toField (required): Google metadata target field (title, objectType, mimeType, updateTime, createTime, containerName, contentLanguage, sourceRepositoryUrl, hash, keywords, searchQualityMetadata).
  • defaultValue (optional): fallback when fromField is missing or blank.
  • keepFromField (optional): when true, keeps fromField in structured data; otherwise it is excluded.

Reference list (Google): https://developers.google.com/workspace/cloud-search/docs/reference/rest/v1/indexing.datasources.items#ItemMetadata

Two ItemMetadata fields are not supported by a mapping: contextAttributes and interactions. Both are lists of nested objects rather than a single value derived from one crawled field, so they don't fit this mapping model.

When a mapping does not produce a value, committer defaults are applied where available (objectType=document, mimeType from detected content type, sourceRepositoryUrl from document reference).

queue​

Configures the committer queue used to batch documents before sending them to the target system. Adjust queue settings to tune throughput and memory usage.

requestMode​

Priority/consistency mode sent with indexing and delete requests. Use ASYNCHRONOUS (default) for connector-style full or incremental crawls with higher throughput, or SYNCHRONOUS for latency-sensitive, interactive requests that are more strictly rate-limited by Google Cloud Search.

Allowed Values​

  • SYNCHRONOUS
  • ASYNCHRONOUS

restrictions​

Defines filter conditions that restrict which documents this committer processes. Documents not matching the restrictions are ignored by this committer.

secretKeyPath​

Path to the Google service account JSON key file, with access to the Cloud Search indexing API.

sourceIdField​

Metadata field holding the value to use as the document unique identifier. Defaults to the document reference.

structuredDataMappings​

Declares the Google Cloud Search structured data value type to use for individual metadata fields sent as structured data. Each mapping supports:

  • field (required): the metadata/structured-data field name.
  • type (optional, default TEXT): one of TEXT, INTEGER, DOUBLE, DATE, TIMESTAMP, BOOLEAN, ENUM, HTML.

Fields without a matching mapping are always sent as TEXT, which is safe regardless of how the property is declared in your Cloud Search data source schema. The declared type must match what your schema actually registers for that property; this committer cannot verify that on its own. Values that fail to parse as the declared type are logged as a warning and sent as text instead of failing the batch.

Supported type values:

  • TEXT — any string (the default).
  • INTEGER — whole number, e.g. 123.
  • DOUBLE — decimal number, e.g. 1.5.
  • DATE — calendar date, no time: 2026-07-14.
  • TIMESTAMP — date and time (RFC-3339): 2026-07-14T12:30:00Z.
  • BOOLEAN — true or false.
  • ENUM — any string (see the 32-value cap below).
  • HTML — a string containing HTML markup.

For date-like fields: use DATE when only the calendar day matters (e.g., a publish date), and TIMESTAMP when the time itself matters (e.g., a last-crawled instant).

Google Cloud Search also caps ENUM properties at 32 values per item; a field mapped as ENUM that exceeds this falls back to text with a warning rather than failing the whole batch.

Multi-valued fields: nothing special needed on this committer's side. Every metadata field already holds a list of values by default, and whichever type is declared, the committer sends the complete list of values to Google for that field — no extra configuration required. Whether Google accepts multiple values for a given property depends on your data source's schema (specifically, that property's repeatable setting) — for that side, check Google's own documentation on repeatable properties, since that's configured and enforced entirely by Google, not something this committer can see or control. One exception on this committer's side: BOOLEAN values are always sent as a single value (Google's API models booleans as non-repeatable), so if a repeatable field is mapped as type="BOOLEAN", only the first value collected is sent.

uploadFormat​

The format used to upload document content: "RAW" (the original, unparsed content, captured ahead of time into a metadata field, e.g., using a BinaryContentTransformer pre-parse handler) or "TEXT" (the extracted plain-text content). Default is "RAW".

Allowed Values​

  • RAW
  • TEXT