Package com.norconex.importer.handler.condition.impl

Class DomConditionConfig

java.lang.Object
com.norconex.importer.handler.BaseLabelableConfig
com.norconex.importer.handler.condition.BaseConditionConfig
com.norconex.importer.handler.condition.impl.DomConditionConfig
All Implemented Interfaces:
Labelable
public class DomConditionConfig extends BaseConditionConfig

A condition using a Document Object Model (DOM) representation of an HTML, XHTML, or XML document content to match an element, attribute or value.

In order to construct a DOM tree, text is loaded entirely into memory. It uses the document content to create the DOM by default, but it can also use metadata fields. If more than one metadata field values are identified as the source of DOM content, only one needs to match for this condition to be true. Use this condition with caution if you know you'll need to parse huge files. You can use TextFilter instead if this is a concern.

The jsoup parser library is used to load the content into a DOM tree. Elements are referenced using a CSS or JQuery-like syntax.

The use of a value matcher is optional. Without one, any element found by the provided DOM selector will constitute a match. If both a DOM selector and a value matcher are provided, the matching selector element value(s) will be retrieved and the value matcher will be applied against it (or them) for a match.

It is possible to control what gets extracted exactly for matching purposes thanks to the "extract" argument of the new method setExtract(String). Possible values are:

DomUtil.getElementValue(org.jsoup.nodes.Element,java.lang.String)

Should be used as a pre-parse handler.

Content-types

If you are dealing with multiple document types and you are using this condition on the document content, it is important to restrict this condition to text-based XML-like content only to prevent DOM-parsing errors.

By default this condition only applies to documents matching the content types listed in CommonMatchers.DOM_CONTENT_TYPES. Other content types always make this condition false.

You can overwrite these default content types by providing your own content type matcher. Make sure the content types you use represent a file with HTML or XML-like markup tags.

Character encoding

When used as a pre-parse handler, this condition uses the detected character encoding unless the character encoding was specified using setSourceCharset(Charset). Since document parsing should always converts content to UTF-8, UTF-8 is always assumed when used as a post-parse handler.

XML vs HTML

You can specify which DOM parser to use when reading documents. The default is "html" and will try to normalize/fix the content as HTML. This is generally a desired behavior, but this can sometimes have your selector fail. If you encounter this problem, try switching to "xml" parser, which does not attempt normalization on the content. The drawback with "xml" is you may not get all HTML-specific selector options to work. If you know you are dealing with XML to begin with, specifying "xml" is a good option.

See Also:

  • Constructor Details

    • DomConditionConfig

      public DomConditionConfig()

  • Method Details

    • getFieldMatcher

      public TextMatcher getFieldMatcher()
      Gets this filter field matcher.
      Returns:
      field matcher

    • setFieldMatcher

      public DomConditionConfig setFieldMatcher(TextMatcher fieldMatcher)
      Sets this condition field matcher.
      Parameters:
      fieldMatcher - field matcher

    • getValueMatcher

      public TextMatcher getValueMatcher()
      Gets this condition value matcher.
      Returns:
      value matcher

    • setValueMatcher

      public DomConditionConfig setValueMatcher(TextMatcher valueMatcher)
      Sets this condition value matcher.
      Parameters:
      valueMatcher - value matcher

    • getContentTypeMatcher

      public TextMatcher getContentTypeMatcher()
      Gets this condition content-type matcher.
      Returns:
      content-type matcher

    • setContentTypeMatcher

      public DomConditionConfig setContentTypeMatcher(TextMatcher contentTypeMatcher)
      Sets this condition content-type matcher.
      Parameters:
      contentTypeMatcher - content-type matcher

    • getExtract

      public String getExtract()
      Gets what should be extracted for the value. One of "text" (default), "html", or "outerHtml". null means this class will use the default ("text").
      Returns:
      what should be extracted for the value

    • setExtract

      public DomConditionConfig setExtract(String extract)
      Sets what should be extracted for the value. One of "text" (default), "html", or "outerHtml". null means this class will use the default ("text").
      Parameters:
      extract - what should be extracted for the value

    • getParser

      public String getParser()
      Gets the parser to use when creating the DOM-tree.
      Returns:
      html (default) or xml.

    • setParser

      public DomConditionConfig setParser(String parser)
      Sets the parser to use when creating the DOM-tree.
      Parameters:
      parser - html or xml.

    • getSelector

      public String getSelector()

    • setSelector

      public DomConditionConfig setSelector(String selector)

    • getSourceCharset

      public Charset getSourceCharset()
      The presumed source character encoding. Usually ignored and presumed to be UTF-8 if the document has been parsed already.
      Returns:
      character encoding of the source to be transformed

    • setSourceCharset

      public DomConditionConfig setSourceCharset(Charset sourceCharset)
      The presumed source character encoding. Usually ignored and presumed to be UTF-8 if the document has been parsed already.
      Parameters:
      sourceCharset - character encoding of the source to be transformed
      Returns:
      this.

    • equals

      public boolean equals(Object o)
      Overrides:
      equals in class BaseLabelableConfig

    • canEqual

      protected boolean canEqual(Object other)
      Overrides:
      canEqual in class BaseLabelableConfig

    • hashCode

      public int hashCode()
      Overrides:
      hashCode in class BaseLabelableConfig

    • toString

      public String toString()
      Overrides:
      toString in class BaseLabelableConfig