TextChunkSplitter
Splits a document's text into chunks sized for embedding models.
Notes
Runs on the document's extracted text, so the same configuration chunks a PDF, an HTML page, or a Word document alike — no per-format splitter needed. Should be used as a post-parse handler.
Each chunk is capped at maxChunkSize characters, overlap included, but the
cut point within that limit is chosen to avoid breaking content awkwardly: the
last paragraph break is preferred, falling back to the last sentence, then the
last word, and only cutting mid-word if none of those are found.
Consecutive chunks can repeat a few trailing characters of the previous one
via chunkOverlap, so a passage spanning a chunk boundary is not lost from
either side's context. The repeated characters count toward maxChunkSize,
so a chunk never goes over it, and chunkOverlap cannot be more than half of
maxChunkSize: past that, most of each chunk would be a repeat. A configuration
that does not follow this is rejected when the importer starts.
A document short enough to fit in a single chunk (maxChunkSize characters or
fewer) is left untouched — no child documents are created, and the original
passes through as-is. This applies to chunks too: they are documents of their
own, going through the importer again from its first handler, and are not
split a second time.
Each chunk produced carries a document.chunk.index field (1-based) and a
document.chunk.count field with the total number of chunks. Each chunk is a
document of its own, and is processed as such right after the split: it goes
through the importer again from its first handler and is committed (or
rejected) like any other document, whatever becomes of the original.
Unlike other splitters, the original document is discarded by default once it
has been split: a document long enough to need chunking is too long to embed
whole, so passing it along next to its chunks would make the embedding step
fail. The other splitters keep it unless told to discardOriginal; this one
has keepOriginal instead, false by default, so that no boolean property of
the crawler's configuration defaults to true. Set keepOriginal to true if
you do want to keep the original.
Examples
The following example chunks documents into roughly 500-character pieces
with 50 characters of overlap between consecutive chunks, ready to be
embedded by TextEmbeddingTransformer.
- YAML
- JSON
- XML
handler:
class: TextChunkSplitter
maxChunkSize: 500
chunkOverlap: 50
{
"handler": {
"class": "TextChunkSplitter",
"maxChunkSize": 500,
"chunkOverlap": 50
}
}
<handler>
<class>TextChunkSplitter</class>
<maxChunkSize>500</maxChunkSize>
<chunkOverlap>50</chunkOverlap>
</handler>
Usage
Full configuration skeleton, with every option and its default
- YAML
- JSON
- XML
class: TextChunkSplitter
chunkOverlap: 0
keepOriginal: false
label: string
maxChunkSize: 0
referenceChunkPrefix: string
{
"class": "TextChunkSplitter",
"chunkOverlap": 0,
"keepOriginal": false,
"label": "string",
"maxChunkSize": 0,
"referenceChunkPrefix": "string"
}
<handler>
<class>TextChunkSplitter</class>
<chunkOverlap>0</chunkOverlap>
<keepOriginal>false</keepOriginal>
<label>string</label>
<maxChunkSize>0</maxChunkSize>
<referenceChunkPrefix>string</referenceChunkPrefix>
</handler>
Properties
| Property | Type | Required | Default |
|---|---|---|---|
chunkOverlap | integer | No | 100 |
keepOriginal | boolean | No | false |
label | string | No | - |
maxChunkSize | integer | No | 1000 |
referenceChunkPrefix | string | No | #chunk |
Property Details
chunkOverlap
The number of characters from the end of a chunk to repeat at the start of
the next one. They count toward maxChunkSize, and must not be more than half
of it. Default is 100, so a maxChunkSize under 200 needs a smaller overlap
set as well. Set to 0 to disable.
keepOriginal
Keep the original document, next to its chunks, after it was split. Only
applies when the document was actually split. Default is false: the
original is discarded in favor of its chunks. This is the reverse of the other
splitters (which have discardOriginal, false by default), for the reason
given above.
label
An optional, user-supplied label for this step in the importer pipeline. It has no effect on processing — it exists purely to help identify this step, for example in large configurations or in the visual configurator.
maxChunkSize
The maximum number of characters a chunk should have, the overlap with the previous chunk included. A chunk may be shorter when a clean break (paragraph, sentence, or word) is found before this limit. Must be at least 1; anything lower is rejected. Default is 1000.
referenceChunkPrefix
Defines the prefix added before the chunk number in a document reference. Blank means the default, "#chunk", so chunk references never collide with the reference of a real document. For example: