Skip to main content

TextChunkSplitter

Splits a document's text into chunks sized for embedding models.

Notes​

Runs on the document's extracted text, so the same configuration chunks a PDF, an HTML page, or a Word document alike — no per-format splitter needed. Should be used as a post-parse handler.

Each chunk is capped at maxChunkSize characters, overlap included, but the cut point within that limit is chosen to avoid breaking content awkwardly: the last paragraph break is preferred, falling back to the last sentence, then the last word, and only cutting mid-word if none of those are found.

Consecutive chunks can repeat a few trailing characters of the previous one via chunkOverlap, so a passage spanning a chunk boundary is not lost from either side's context. The repeated characters count toward maxChunkSize, so a chunk never goes over it, and chunkOverlap cannot be more than half of maxChunkSize: past that, most of each chunk would be a repeat. A configuration that does not follow this is rejected when the importer starts.

A document short enough to fit in a single chunk (maxChunkSize characters or fewer) is left untouched — no child documents are created, and the original passes through as-is. This applies to chunks too: they are documents of their own, going through the importer again from its first handler, and are not split a second time.

Each chunk produced carries a document.chunk.index field (1-based) and a document.chunk.count field with the total number of chunks. Each chunk is a document of its own, and is processed as such right after the split: it goes through the importer again from its first handler and is committed (or rejected) like any other document, whatever becomes of the original.

Unlike other splitters, the original document is discarded by default once it has been split: a document long enough to need chunking is too long to embed whole, so passing it along next to its chunks would make the embedding step fail. The other splitters keep it unless told to discardOriginal; this one has keepOriginal instead, false by default, so that no boolean property of the crawler's configuration defaults to true. Set keepOriginal to true if you do want to keep the original.

Examples​

The following example chunks documents into roughly 500-character pieces with 50 characters of overlap between consecutive chunks, ready to be embedded by TextEmbeddingTransformer.

handler:
class: TextChunkSplitter
maxChunkSize: 500
chunkOverlap: 50

Usage​

Full configuration skeleton, with every option and its default
class: TextChunkSplitter
chunkOverlap: 0
keepOriginal: false
label: string
maxChunkSize: 0
referenceChunkPrefix: string

Properties​

PropertyTypeRequiredDefault
chunkOverlapintegerNo100
keepOriginalbooleanNofalse
labelstringNo-
maxChunkSizeintegerNo1000
referenceChunkPrefixstringNo#chunk

Property Details​

chunkOverlap​

The number of characters from the end of a chunk to repeat at the start of the next one. They count toward maxChunkSize, and must not be more than half of it. Default is 100, so a maxChunkSize under 200 needs a smaller overlap set as well. Set to 0 to disable.

keepOriginal​

Keep the original document, next to its chunks, after it was split. Only applies when the document was actually split. Default is false: the original is discarded in favor of its chunks. This is the reverse of the other splitters (which have discardOriginal, false by default), for the reason given above.

label​

An optional, user-supplied label for this step in the importer pipeline. It has no effect on processing — it exists purely to help identify this step, for example in large configurations or in the visual configurator.

maxChunkSize​

The maximum number of characters a chunk should have, the overlap with the previous chunk included. A chunk may be shorter when a clean break (paragraph, sentence, or word) is found before this limit. Must be at least 1; anything lower is rejected. Default is 1000.

referenceChunkPrefix​

Defines the prefix added before the chunk number in a document reference. Blank means the default, "#chunk", so chunk references never collide with the reference of a real document. For example: