Package com.norconex.importer.handler.splitter.impl

Class DomSplitterConfig

java.lang.Object
com.norconex.importer.handler.BaseLabelableConfig
com.norconex.importer.handler.BaseDocHandlerConfig
com.norconex.importer.handler.splitter.BaseDocumentSplitterConfig
com.norconex.importer.handler.splitter.impl.DomSplitterConfig
All Implemented Interfaces:
Labelable
public class DomSplitterConfig extends BaseDocumentSplitterConfig

Splits HTML, XHTML, or XML document on elements matching a given selector.

This class constructs a DOM tree from the document content. That DOM tree is loaded entirely into memory. Use this splitter with caution if you know you'll need to parse huge files. It may be preferable to use a stream-based approach if this is a concern (e.g., XmlStreamSplitter).

The jsoup parser library is used to load a document content into a DOM tree. Elements are referenced using a CSS or JQuery-like syntax.

Should be used as a pre-parse handler.

Content-types

By default, this filter is restricted to (applies only to) documents matching the restrictions returned by CommonRestrictions.domContentTypes(String). You can specify your own content types if you know they represent a file with HTML or XML-like markup tags.

Since 2.5.0, when used as a pre-parse handler, this class attempts to detect the content character encoding unless the character encoding was specified using setSourceCharset(Charset). Since document parsing converts content to UTF-8, UTF-8 is always assumed when used as a post-parse handler.

Since 2.8.0, you can specify which parser to use when reading documents. The default is "html" and will normalize the content as HTML. This is generally a desired behavior, but this can sometimes have your selector fail. If you encounter this problem, try switching to "xml" parser, which does not attempt normalization on the content. The drawback with "xml" is you may not get all HTML-specific selector options to work. If you know you are dealing with XML to begin with, specifying "xml" should be a good option.

See Also:

  • Constructor Details

    • DomSplitterConfig

      public DomSplitterConfig()

  • Method Details

    • setFieldMatcher

      public DomSplitterConfig setFieldMatcher(TextMatcher fieldMatcher)

    • setContentTypeMatcher

      public DomSplitterConfig setContentTypeMatcher(TextMatcher contentTypeMatcher)
      The matcher of content types to apply splitting on. No attempt to split documents of any other content types will be made. Default is CommonMatchers.DOM_CONTENT_TYPES.
      Parameters:
      contentTypeMatcher - content type matcher
      Returns:
      this

    • getSelector

      public String getSelector()
      The CSS-like selector (see class documentation) identifying which elements to split.
      Returns:
      the selector

    • getSourceCharset

      public Charset getSourceCharset()
      The presumed source character set.
      Returns:
      character set of the source to be transformed

    • getParser

      public String getParser()
      The type of parser to use when creating the DOM-tree. Default is html.
      Returns:
      html or xml.

    • getFieldMatcher

      public TextMatcher getFieldMatcher()
      Matcher of one or more fields to use as the source of content to split into new documents, instead of the original document content.
      Returns:
      field matcher

    • getContentTypeMatcher

      public TextMatcher getContentTypeMatcher()
      The matcher of content types to apply splitting on. No attempt to split documents of any other content types will be made. Default is CommonMatchers.DOM_CONTENT_TYPES.
      Returns:
      content type matcher

    • setSelector

      public DomSplitterConfig setSelector(String selector)
      The CSS-like selector (see class documentation) identifying which elements to split.
      Parameters:
      selector - the selector
      Returns:
      this.

    • setSourceCharset

      public DomSplitterConfig setSourceCharset(Charset sourceCharset)
      The presumed source character set.
      Parameters:
      sourceCharset - character set of the source to be transformed
      Returns:
      this.

    • setParser

      public DomSplitterConfig setParser(String parser)
      The type of parser to use when creating the DOM-tree. Default is html.
      Parameters:
      parser - html or xml.
      Returns:
      this.

    • equals

      public boolean equals(Object o)
      Overrides:
      equals in class BaseDocumentSplitterConfig

    • canEqual

      protected boolean canEqual(Object other)
      Overrides:
      canEqual in class BaseDocumentSplitterConfig

    • hashCode

      public int hashCode()
      Overrides:
      hashCode in class BaseDocumentSplitterConfig

    • toString

      public String toString()
      Overrides:
      toString in class BaseDocumentSplitterConfig