Package com.norconex.importer.handler.splitter.impl

Class TextChunkSplitterConfig

java.lang.Object
com.norconex.importer.handler.BaseLabelableConfig
com.norconex.importer.handler.BaseDocHandlerConfig
com.norconex.importer.handler.splitter.BaseDocumentSplitterConfig
com.norconex.importer.handler.splitter.impl.TextChunkSplitterConfig
All Implemented Interfaces:
Labelable
public class TextChunkSplitterConfig extends BaseDocumentSplitterConfig

Splits a document's text into chunks sized for embedding models, breaking at the cleanest boundary available (paragraph, then sentence, then word) rather than cutting mid-word at a fixed offset.

See Also:

  • Field Details

  • Constructor Details

    • TextChunkSplitterConfig

      public TextChunkSplitterConfig()

  • Method Details

    • getMaxChunkSize

      public int getMaxChunkSize()
      The maximum number of characters a chunk should have. A chunk may be shorter when a clean break (paragraph, sentence, or word) is found before this limit. Must be at least 1. Default is 1000.

    • getChunkOverlap

      public int getChunkOverlap()
      The number of characters from the end of a chunk to repeat at the start of the next one, so a passage spanning a chunk boundary is not lost from either side's context. Default is 100. Set to 0 to disable.

    • getReferenceChunkPrefix

      public String getReferenceChunkPrefix()
      String to append to the parent document reference to form each chunk's reference, followed by the chunk number (1-based). Blank means the default, "#chunk".

    • setMaxChunkSize

      public TextChunkSplitterConfig setMaxChunkSize(int maxChunkSize)
      The maximum number of characters a chunk should have. A chunk may be shorter when a clean break (paragraph, sentence, or word) is found before this limit. Must be at least 1. Default is 1000.
      Returns:
      this.

    • setChunkOverlap

      public TextChunkSplitterConfig setChunkOverlap(int chunkOverlap)
      The number of characters from the end of a chunk to repeat at the start of the next one, so a passage spanning a chunk boundary is not lost from either side's context. Default is 100. Set to 0 to disable.
      Returns:
      this.

    • setReferenceChunkPrefix

      public TextChunkSplitterConfig setReferenceChunkPrefix(String referenceChunkPrefix)
      String to append to the parent document reference to form each chunk's reference, followed by the chunk number (1-based). Blank means the default, "#chunk".
      Returns:
      this.

    • equals

      public boolean equals(Object o)
      Overrides:
      equals in class BaseDocumentSplitterConfig

    • canEqual

      protected boolean canEqual(Object other)
      Overrides:
      canEqual in class BaseDocumentSplitterConfig

    • hashCode

      public int hashCode()
      Overrides:
      hashCode in class BaseDocumentSplitterConfig

    • toString

      public String toString()
      Overrides:
      toString in class BaseDocumentSplitterConfig