Class TextChunkSplitter

java.lang.Object
com.norconex.importer.handler.splitter.impl.TextChunkSplitter
All Implemented Interfaces:
Configurable<TextChunkSplitterConfig>, ConfigurableDocHandler<TextChunkSplitterConfig>, DocHandler

public class TextChunkSplitter extends Object implements ConfigurableDocHandler<TextChunkSplitterConfig>

Splits a document's text into chunks sized for embedding models. Meant to run post-parse, on already-extracted plain text, so the same configuration chunks a PDF, an HTML page, or a Word document alike, instead of every format needing its own splitter.

Each chunk, overlap included, is capped at TextChunkSplitterConfig.getMaxChunkSize() characters, but the cut point within that limit is chosen to avoid breaking content awkwardly: the last paragraph break is preferred, falling back to the last sentence, then the last word, and only cutting mid-word if none of those are found (see TextReader). Consecutive chunks can repeat a few trailing characters of the previous one via TextChunkSplitterConfig.getChunkOverlap(), so a passage spanning a chunk boundary is not lost from either side's context.

A document short enough to fit in a single chunk is left untouched (no child documents are created). A document that is split is discarded in favor of its chunks, unless TextChunkSplitterConfig.isKeepOriginal() is turned on (the reverse of what the other splitters do by default: see that class for why).

See Also:
  • Field Details

  • Constructor Details

    • TextChunkSplitter

      public TextChunkSplitter()
  • Method Details

    • init

      public void init() throws IOException
      Description copied from interface: DocHandler
      Initialize any required resources. Invoked once per Importer instance, by the Importer.
      Specified by:
      init in interface DocHandler
      Throws:
      IOException - An I/O exception of some sort has occurred
    • handle

      public boolean handle(DocHandlerContext docCtx) throws IOException
      Description copied from interface: DocHandler
      Perform any operations on a document.
      Specified by:
      handle in interface DocHandler
      Parameters:
      docCtx - the document handling context
      Returns:
      false to stop processing and reject a document or true to proceed to the next configured handler.
      Throws:
      IOException - An I/O exception of some sort has occurred
    • split

      public void split(DocHandlerContext docCtx) throws DocHandlerException
      Cuts the document's text into chunks, added to the context as child documents. Does nothing if the text fits in one chunk.
      Parameters:
      docCtx - the document to split
      Throws:
      DocHandlerException - if the text could not be read, or the configuration cannot be used
    • getConfiguration

      public TextChunkSplitterConfig getConfiguration()
      Specified by:
      getConfiguration in interface Configurable<TextChunkSplitterConfig>
    • equals

      public boolean equals(Object o)
      Overrides:
      equals in class Object
    • canEqual

      protected boolean canEqual(Object other)
    • hashCode

      public int hashCode()
      Overrides:
      hashCode in class Object
    • toString

      public String toString()
      Overrides:
      toString in class Object