Class TextChunkSplitter
- All Implemented Interfaces:
Configurable<TextChunkSplitterConfig>,ConfigurableDocHandler<TextChunkSplitterConfig>,DocHandler
Splits a document's text into chunks sized for embedding models. Meant to run post-parse, on already-extracted plain text, so the same configuration chunks a PDF, an HTML page, or a Word document alike, instead of every format needing its own splitter.
Each chunk, overlap included, is capped at
TextChunkSplitterConfig.getMaxChunkSize()
characters, but the cut point within that limit is chosen to avoid
breaking content awkwardly: the last paragraph break is preferred, falling
back to the last sentence, then the last word, and only cutting mid-word
if none of those are found (see
TextReader). Consecutive chunks can
repeat a few trailing characters of the previous one via
TextChunkSplitterConfig.getChunkOverlap(), so a passage spanning
a chunk boundary is not lost from either side's context.
A document short enough to fit in a single chunk is left untouched (no
child documents are created). A document that is split is discarded in
favor of its chunks, unless
TextChunkSplitterConfig.isKeepOriginal() is turned on (the reverse
of what the other splitters do by default: see that class for why).
- See Also:
-
Field Summary
FieldsFields inherited from interface com.norconex.commons.lang.config.Configurable
PROPERTYFields inherited from interface com.norconex.importer.handler.DocHandler
NAME -
Constructor Summary
Constructors -
Method Summary
Modifier and TypeMethodDescriptionprotected booleanbooleanbooleanhandle(DocHandlerContext docCtx) Perform any operations on a document.inthashCode()voidinit()Initialize any required resources.voidsplit(DocHandlerContext docCtx) Cuts the document's text into chunks, added to the context as child documents.toString()Methods inherited from class java.lang.Object
clone, finalize, getClass, notify, notifyAll, wait, wait, waitMethods inherited from interface com.norconex.importer.handler.DocHandler
close
-
Field Details
-
DOC_CHUNK_INDEX
- See Also:
-
DOC_CHUNK_COUNT
- See Also:
-
-
Constructor Details
-
TextChunkSplitter
public TextChunkSplitter()
-
-
Method Details
-
init
Description copied from interface:DocHandlerInitialize any required resources. Invoked once per Importer instance, by the Importer.- Specified by:
initin interfaceDocHandler- Throws:
IOException- An I/O exception of some sort has occurred
-
handle
Description copied from interface:DocHandlerPerform any operations on a document.- Specified by:
handlein interfaceDocHandler- Parameters:
docCtx- the document handling context- Returns:
falseto stop processing and reject a document ortrueto proceed to the next configured handler.- Throws:
IOException- An I/O exception of some sort has occurred
-
split
Cuts the document's text into chunks, added to the context as child documents. Does nothing if the text fits in one chunk.- Parameters:
docCtx- the document to split- Throws:
DocHandlerException- if the text could not be read, or the configuration cannot be used
-
getConfiguration
- Specified by:
getConfigurationin interfaceConfigurable<TextChunkSplitterConfig>
-
equals
-
canEqual
-
hashCode
public int hashCode() -
toString
-