Package com.norconex.importer.handler.splitter.impl

Class TextChunkSplitter

java.lang.Object
com.norconex.importer.handler.splitter.AbstractDocumentSplitter<TextChunkSplitterConfig>
com.norconex.importer.handler.splitter.impl.TextChunkSplitter
All Implemented Interfaces:
Configurable<TextChunkSplitterConfig>, ConfigurableDocHandler<TextChunkSplitterConfig>, DocHandler
public class TextChunkSplitter extends AbstractDocumentSplitter<TextChunkSplitterConfig>

Splits a document's text into chunks sized for embedding models. Meant to run post-parse, on already-extracted plain text, so the same configuration chunks a PDF, an HTML page, or a Word document alike, instead of every format needing its own splitter.

Each chunk is capped at TextChunkSplitterConfig.getMaxChunkSize() characters, but the cut point within that limit is chosen to avoid breaking content awkwardly: the last paragraph break is preferred, falling back to the last sentence, then the last word, and only cutting mid-word if none of those are found (see TextReader). Consecutive chunks can repeat a few trailing characters of the previous one via TextChunkSplitterConfig.getChunkOverlap(), so a passage spanning a chunk boundary is not lost from either side's context.

A document short enough to fit in a single chunk is left untouched (no child documents are created). A document that is split is discarded in favor of its chunks, unless BaseDocumentSplitterConfig.isDiscardOriginal() is turned off.

See Also: