Class TextChunkSplitterConfig
- All Implemented Interfaces:
Labelable
Splits a document's text into chunks sized for embedding models, breaking at the cleanest boundary available (paragraph, then sentence, then word) rather than cutting mid-word at a fixed offset.
An exception among splitters. The other splitters extend
BaseDocumentSplitterConfig
and keep the original document unless told to discard it
(discardOriginal, false by default). This one does the
opposite: the original is discarded once it has been split, because a
document long enough to be chunked is too long for the embedding step that
follows, so passing it on alongside its chunks would only make that step
fail. Rather than a discardOriginal that is true by
default, which would make this the one boolean of the crawler's
configuration not to default to false, it has isKeepOriginal(),
false by default. That is why this class does not extend
BaseDocumentSplitterConfig.
- See Also:
-
Field Summary
Fields -
Constructor Summary
Constructors -
Method Summary
Modifier and TypeMethodDescriptionprotected booleanbooleanintThe number of characters from the end of a chunk to repeat at the start of the next one, so a passage spanning a chunk boundary is not lost from either side's context.intThe maximum number of characters a chunk should have, the overlap with the previous chunk included.String to append to the parent document reference to form each chunk's reference, followed by the chunk number (1-based).inthashCode()booleanWhether to keep the original document, next to its chunks, once it was split.setChunkOverlap(int chunkOverlap) The number of characters from the end of a chunk to repeat at the start of the next one, so a passage spanning a chunk boundary is not lost from either side's context.setKeepOriginal(boolean keepOriginal) Whether to keep the original document, next to its chunks, once it was split.setMaxChunkSize(int maxChunkSize) The maximum number of characters a chunk should have, the overlap with the previous chunk included.setReferenceChunkPrefix(String referenceChunkPrefix) String to append to the parent document reference to form each chunk's reference, followed by the chunk number (1-based).toString()Methods inherited from class com.norconex.importer.handler.BaseLabelableConfig
getLabel, setLabel
-
Field Details
-
DEFAULT_MAX_CHUNK_SIZE
public static final int DEFAULT_MAX_CHUNK_SIZE- See Also:
-
DEFAULT_CHUNK_OVERLAP
public static final int DEFAULT_CHUNK_OVERLAP- See Also:
-
DEFAULT_REFERENCE_CHUNK_PREFIX
- See Also:
-
-
Constructor Details
-
TextChunkSplitterConfig
public TextChunkSplitterConfig()
-
-
Method Details
-
getMaxChunkSize
public int getMaxChunkSize()The maximum number of characters a chunk should have, the overlap with the previous chunk included. A chunk may be shorter when a clean break (paragraph, sentence, or word) is found before this limit. Must be at least 1. Default is 1000. -
getChunkOverlap
public int getChunkOverlap()The number of characters from the end of a chunk to repeat at the start of the next one, so a passage spanning a chunk boundary is not lost from either side's context. The overlap counts towardgetMaxChunkSize(), and must not be more than half of it. Default is 100. Set to 0 to disable. -
getReferenceChunkPrefix
String to append to the parent document reference to form each chunk's reference, followed by the chunk number (1-based). Blank means the default, "#chunk". -
isKeepOriginal
public boolean isKeepOriginal()Whether to keep the original document, next to its chunks, once it was split. Only applies if the document was actually split: one short enough to fit in a single chunk is never split and always passes through as it is.By default the original is not kept: it is discarded in favor of its chunks. This is the reverse of what the other splitters do (see this class' description for why), and is why the property is a "keep" that is
falseby default, and not a "discard" that would betrue.- Returns:
truewhen keeping the original document after an effective split,false(the default) when discarding it
-
setMaxChunkSize
The maximum number of characters a chunk should have, the overlap with the previous chunk included. A chunk may be shorter when a clean break (paragraph, sentence, or word) is found before this limit. Must be at least 1. Default is 1000.- Returns:
this.
-
setChunkOverlap
The number of characters from the end of a chunk to repeat at the start of the next one, so a passage spanning a chunk boundary is not lost from either side's context. The overlap counts towardgetMaxChunkSize(), and must not be more than half of it. Default is 100. Set to 0 to disable.- Returns:
this.
-
setReferenceChunkPrefix
String to append to the parent document reference to form each chunk's reference, followed by the chunk number (1-based). Blank means the default, "#chunk".- Returns:
this.
-
setKeepOriginal
Whether to keep the original document, next to its chunks, once it was split. Only applies if the document was actually split: one short enough to fit in a single chunk is never split and always passes through as it is.By default the original is not kept: it is discarded in favor of its chunks. This is the reverse of what the other splitters do (see this class' description for why), and is why the property is a "keep" that is
falseby default, and not a "discard" that would betrue.- Parameters:
keepOriginal-trueto keep the original document after an effective split- Returns:
this.
-
equals
- Overrides:
equalsin classBaseLabelableConfig
-
canEqual
- Overrides:
canEqualin classBaseLabelableConfig
-
hashCode
public int hashCode()- Overrides:
hashCodein classBaseLabelableConfig
-
toString
- Overrides:
toStringin classBaseLabelableConfig
-