Class TextChunkSplitterConfig

All Implemented Interfaces:
Labelable

public class TextChunkSplitterConfig extends BaseDocHandlerConfig

Splits a document's text into chunks sized for embedding models, breaking at the cleanest boundary available (paragraph, then sentence, then word) rather than cutting mid-word at a fixed offset.

An exception among splitters. The other splitters extend BaseDocumentSplitterConfig and keep the original document unless told to discard it (discardOriginal, false by default). This one does the opposite: the original is discarded once it has been split, because a document long enough to be chunked is too long for the embedding step that follows, so passing it on alongside its chunks would only make that step fail. Rather than a discardOriginal that is true by default, which would make this the one boolean of the crawler's configuration not to default to false, it has isKeepOriginal(), false by default. That is why this class does not extend BaseDocumentSplitterConfig.

See Also:
  • Field Details

  • Constructor Details

    • TextChunkSplitterConfig

      public TextChunkSplitterConfig()
  • Method Details

    • getMaxChunkSize

      public int getMaxChunkSize()
      The maximum number of characters a chunk should have, the overlap with the previous chunk included. A chunk may be shorter when a clean break (paragraph, sentence, or word) is found before this limit. Must be at least 1. Default is 1000.
    • getChunkOverlap

      public int getChunkOverlap()
      The number of characters from the end of a chunk to repeat at the start of the next one, so a passage spanning a chunk boundary is not lost from either side's context. The overlap counts toward getMaxChunkSize(), and must not be more than half of it. Default is 100. Set to 0 to disable.
    • getReferenceChunkPrefix

      public String getReferenceChunkPrefix()
      String to append to the parent document reference to form each chunk's reference, followed by the chunk number (1-based). Blank means the default, "#chunk".
    • isKeepOriginal

      public boolean isKeepOriginal()
      Whether to keep the original document, next to its chunks, once it was split. Only applies if the document was actually split: one short enough to fit in a single chunk is never split and always passes through as it is.

      By default the original is not kept: it is discarded in favor of its chunks. This is the reverse of what the other splitters do (see this class' description for why), and is why the property is a "keep" that is false by default, and not a "discard" that would be true.

      Returns:
      true when keeping the original document after an effective split, false (the default) when discarding it
    • setMaxChunkSize

      public TextChunkSplitterConfig setMaxChunkSize(int maxChunkSize)
      The maximum number of characters a chunk should have, the overlap with the previous chunk included. A chunk may be shorter when a clean break (paragraph, sentence, or word) is found before this limit. Must be at least 1. Default is 1000.
      Returns:
      this.
    • setChunkOverlap

      public TextChunkSplitterConfig setChunkOverlap(int chunkOverlap)
      The number of characters from the end of a chunk to repeat at the start of the next one, so a passage spanning a chunk boundary is not lost from either side's context. The overlap counts toward getMaxChunkSize(), and must not be more than half of it. Default is 100. Set to 0 to disable.
      Returns:
      this.
    • setReferenceChunkPrefix

      public TextChunkSplitterConfig setReferenceChunkPrefix(String referenceChunkPrefix)
      String to append to the parent document reference to form each chunk's reference, followed by the chunk number (1-based). Blank means the default, "#chunk".
      Returns:
      this.
    • setKeepOriginal

      public TextChunkSplitterConfig setKeepOriginal(boolean keepOriginal)
      Whether to keep the original document, next to its chunks, once it was split. Only applies if the document was actually split: one short enough to fit in a single chunk is never split and always passes through as it is.

      By default the original is not kept: it is discarded in favor of its chunks. This is the reverse of what the other splitters do (see this class' description for why), and is why the property is a "keep" that is false by default, and not a "discard" that would be true.

      Parameters:
      keepOriginal - true to keep the original document after an effective split
      Returns:
      this.
    • equals

      public boolean equals(Object o)
      Overrides:
      equals in class BaseLabelableConfig
    • canEqual

      protected boolean canEqual(Object other)
      Overrides:
      canEqual in class BaseLabelableConfig
    • hashCode

      public int hashCode()
      Overrides:
      hashCode in class BaseLabelableConfig
    • toString

      public String toString()
      Overrides:
      toString in class BaseLabelableConfig