Skip to main content

Committers

Send crawled content anywhere

A committer is the last step in the pipeline: it takes each processed document and writes it to its destination. Swap one for another without touching the rest of your configuration. Every name below links to its full configuration reference.

  • 13Stock committers
  • 4Bundled — nothing to install
  • AnyCustom destination, in Java

Search engines

Self-hosted or fully managed — the same crawl feeds either.

  • Elasticsearch and OpenSearchIndex documents into Elasticsearch 7+ or OpenSearch via the REST API.Also covers Amazon OpenSearch Service, Elastic Cloud
  • Apache SolrCommit to Apache Solr or Lucidworks Fusion using the SolrJ client.Also covers Lucidworks Fusion
  • OpenText IDOLIndex content into OpenText IDOL (formerly Micro Focus) via the IDOL API.Also covers Micro Focus
  • Amazon CloudSearchClosed to new AWS customersCommit documents to an Amazon CloudSearch domain.AWS no longer offers CloudSearch to new customers; existing domains keep working, and AWS points new projects at OpenSearch.

Databases and graphs

Land content in a table or as a connected graph.

  • Neo4jStore crawled documents and the relationships between them as nodes in a Neo4j graph.
  • SQL / JDBCCommit to any JDBC-compatible relational database — PostgreSQL, MySQL, Oracle, SQL Server, and the rest.Also covers PostgreSQL, MySQL, Oracle, SQL Server, MariaDB

Streaming

Hand documents to a broker and fan out from there.

  • Apache KafkaPublish crawled documents as messages to Apache Kafka topics, to fan out to whatever consumes them.

Files and debugging

Bundled with every crawler. Nothing to install.

  • JSONWrite documents to a JSON file or directory.
  • XMLWrite documents to XML files.
  • CSVWrite document metadata to CSV files, ready for a spreadsheet or a bulk loader.
  • LogLog each document to the crawler's log output. Ideal for development and testing.

Feeding an AI, RAG, or graph pipeline?

Several of these already land content where AI stacks live. Elasticsearch, OpenSearch, and Solr all support vector search natively, so the same crawl that fills a keyword index fills a hybrid one. Neo4j turns documents and the links between them into a graph. Kafka hands each document to whatever computes your embeddings, and SQL/JDBC stages content in Postgres for a job to pick up. A dedicated vector database — Qdrant, Weaviate, Pinecone, Milvus, Chroma — is a committer away, which is the next section.

What the crawler contributes to a RAG pipeline

Nothing here matches? Write your own.

Extend AbstractBatchCommitter and implement a single method, commitBatch. Queuing, batching, retries, and splitting a failed batch down to individual documents are all handled for you — what's left is the code that talks to your destination.

Read the extension guide

Looking for the other end?

Committers are where crawled content lands. What it can be crawled from — SharePoint, Alfresco and other CMIS repositories, cloud storage, network shares, or any website — is on the Content Sources page.

See what the crawler can read