ApacheKafkaCommitter

Commit documents to Apache Kafka.

committer/apachekafkaJavadoc: ApacheKafkaCommitter

Notes

Commits crawled documents to an Apache Kafka topic. Supports optional topic creation, field mappings to Kafka fields, and configurable partition and replication settings.

Interoperability note:

• ApacheKafkaCommitter provides Kafka-based interoperability out of the box by publishing crawl output to an open event bus.
• This allows downstream integration with a wide range of systems through Kafka-native consumers and connector ecosystems.
• Downstream behavior (for example schema mapping, delete semantics, idempotency, and ACL handling) depends on the target consumer/connector configuration and capabilities.

Examples

The following example uses the minimum required settings to configure Apache Kafka Committer. The Kafka topic won't be created. As such, the topic must already exist in Apache Kafka.

YAML

class: ApacheKafkaCommitter
bootstrapServers: http://some_host:1234
topicName: my-topic

JSON

{
  "class": "ApacheKafkaCommitter",
  "bootstrapServers": "http://some_host:1234",
  "topicName": "my-topic"
}

XML

<committer>
  <class>ApacheKafkaCommitter</class>
  <bootstrapServers>http://some_host:1234</bootstrapServers>
  <topicName>my-topic</topicName>
</committer>

Usage

Full configuration skeleton, with every option and its default

YAML

class: ApacheKafkaCommitter
bootstrapServers: string
createTopic: false
fieldMappings: {}
partitions: 0
queue:
  class: FsQueue
  batchSize: 0
  commitLeftoversOnInit: false
  maxPerFolder: 0
  onCommitFailure:
    ignoreErrors: false
    maxRetries: 0
    retryDelay: 0
    splitBatch: OFF
replicationFactor: 0
restrictions:
  - fieldMatcher: {}
    valueMatcher: {}
topicName: string

JSON

{
  "class": "ApacheKafkaCommitter",
  "bootstrapServers": "string",
  "createTopic": false,
  "fieldMappings": {},
  "partitions": 0,
  "queue": {
    "class": "FsQueue",
    "batchSize": 0,
    "commitLeftoversOnInit": false,
    "maxPerFolder": 0,
    "onCommitFailure": {
      "ignoreErrors": false,
      "maxRetries": 0,
      "retryDelay": 0,
      "splitBatch": "OFF"
    }
  },
  "replicationFactor": 0,
  "restrictions": [
    {
      "fieldMatcher": {},
      "valueMatcher": {}
    }
  ],
  "topicName": "string"
}

XML

<apacheKafkaCommitter>
  <class>ApacheKafkaCommitter</class>
  <bootstrapServers>string</bootstrapServers>
  <createTopic>false</createTopic>
  <fieldMappings/>
  <partitions>0</partitions>
  <queue>
    <class>FsQueue</class>
    <batchSize>0</batchSize>
    <commitLeftoversOnInit>false</commitLeftoversOnInit>
    <maxPerFolder>0</maxPerFolder>
    <onCommitFailure/>
  </queue>
  <replicationFactor>0</replicationFactor>
  <restrictions>
    <restriction>
      <fieldMatcher/>
      <valueMatcher/>
    </restriction>
  </restrictions>
  <topicName>string</topicName>
</apacheKafkaCommitter>

Properties

Property	Type	Required	Default
bootstrapServers	string	No	-
createTopic	boolean	No	false
fieldMappings	Record<string, string>	No	-
partitions	integer	No	0
queue	CommitterQueue	No	FsQueue
replicationFactor	integer	No	0
restrictions	PropertyMatcher[]	No	-
topicName	string	No	-

Property Details

bootstrapServers

A list of host/port pairs in the form host1:port1,host2:port2,... to use for establishing a connection to the Kafka cluster.

createTopic

Create the "Topic Name" in Kafka if it does not exist already.

fieldMappings

Mapping of source fields to target ones (Kafka fields).

partitions

Number of partitions set at topic creation.

queue

Configures the committer queue used to batch documents before sending them to the target system. Adjust queue settings to tune throughput and memory usage.

replicationFactor

Replication factor set at topic creation.

restrictions

Defines filter conditions that restrict which documents this committer processes. Documents not matching the restrictions are ignored by this committer.

topicName

The Kafka topic name to possibly create and use as the target for storing documents.