Skip to main content

DomTransformer

Extract the value of one or more elements or attributes into a target field, or delete matching elements

importerJavadoc: DomTransformer

Notes​

Applies to HTML, XHTML, or XML-like tags document.

This class constructs a DOM tree from a document or field content. That DOM tree is loaded entirely into memory. Use this transformer with caution if you know you'll need to parse huge files. It may be preferable to use RegexTransformer if this is a concern. Also, to help performance and avoid re-creating DOM tree before every DOM extraction you want to perform, try to combine multiple extractions in a single instance of this Transformer.

The jsoup parser library is used to load a document content into a DOM tree. Elements are referenced using a CSS or JQuery-like syntax.

You can control what exactly extract. Define in Operation:

  • text: Default option when extract is blank. The text of the element, including combined children.
  • html: Extracts an element inner HTML (including children).
  • outerHtml: Extracts an element outer HTML (like "html", but includes the "current" tag).
  • ownText: Extracts the text owned by this element only; does not get the combined text of all children.
  • data: Extracts the combined data of a data-element (e.g. <script>).
  • id: Extracts the ID attribute of the element (if any).
  • tagName: Extract the name of the tag of the element.
  • val: Extracts the value of a form element (input, textarea, etc).
  • className: Extracts the literal value of the element's "class" attribute, which may include multiple class names, space separated.
  • cssSelector: Extracts a CSS selector that will uniquely select (identify) this element.
  • attr(attributeKey): Extracts the value of the element attribute matching your replacement for "attributeKey" (e.g. "attr(title)" will extract the "title" attribute).

You can specify a fromField as the source of the HTML to parse instead of using the document content. If multiple values are present for that source field, DOM extraction will be applied to each value.

Dom Operation​

You can specify a defaultValue on each DOM extraction details. When no match occurred for a given selector, the default value will be stored in the toField (as opposed to not storing anything). When matching blanks (see below) you will get an empty string as opposed to the default value. Empty strings and spaces are supported as default values (the default value is now taken literally).

You can set matchBlanks to true to match elements that are present but have blank values. Blank values are empty values or values containing white spaces only. Because white spaces are normalized by the DOM parser, such matches will always return an empty string (spaces will be trimmed). By default elements with blank values are not matched and are ignored.

You can specify which parser to use when reading documents. The default is "html" and will normalize the content as HTML. This is generally a desired behavior, but this can sometimes have your selector fail. If you encounter this problem, try switching to "xml" parser, which does not attempt normalization on the content. The drawback with "xml" is you may not get all HTML-specific selector options to work. If you know you are dealing with XML to begin with, specifying "xml" should be a good option.

Content Deletion from fields​

You can specify whether to delete any elements matched by the selector. You can use with a toField or on its own. Some options are ignored by deletions, such as extract or defaultValue. Because Transformer cannot modify the document content, deletion only applies to metadata fields.

Parser Handler Compatibility​

This functionality can be applied as either a pre-parse or post-parse handler.

When used as a pre-parse handler, it attempts to detect the content character encoding unless the character encoding was specified using sourceCharset. Since document parsing converts content to UTF-8, UTF-8 is always assumed when used as a post-parse handler.

Storing values in an existing field​

If a document already has a target field with the same name, new values are appended to the existing list by default. Use onSet to manage this default behavior.

Content-types​

By default, this filter applies only to documents with content types specified by the CommonRestrictions.domContentTypes(String) method. You can also define your own content types if they represent files containing HTML or XML-like markup tags.

Common content-types:​

  • application/atom+xml
  • application/mathml+xml
  • application/rss+xml
  • application/vnd.wap.xhtml+xml
  • application/x-asp
  • application/xhtml+xml
  • application/xml
  • application/xslt+xml
  • image/svg+xml
  • text/html
  • text/xml

Examples​

Given this HTML snippet...

<div class="firstName">Joe</div>
<div class="lastName">Dalton</div>

The following example will store "Joe" in a "firstName" field and "Dalton" in a "lastName" field.

handler:
  class: DomTransformer
  operations:
    - selector: div.firstName
      toField: firstName
    - selector: div.lastname
      toField: lastName

Usage​

Full configuration skeleton, with every option and its default
class: DomTransformer
contentTypeMatcher:
  ignoreCase: false
  ignoreDiacritic: false
  matchEmpty: false
  method: BASIC
  negateMatches: false
  partial: false
  pattern: string
  replaceAll: false
  trim: false
fieldMatcher:
  ignoreCase: false
  ignoreDiacritic: false
  matchEmpty: false
  method: BASIC
  negateMatches: false
  partial: false
  pattern: string
  replaceAll: false
  trim: false
label: string
operations:
  - defaultValue: string
    delete: false
    extract: string
    matchBlanks: false
    onSet: APPEND
    selector: string
    toField: string
parser: string
sourceCharset: string

Properties​

PropertyTypeRequiredDefault
contentTypeMatcherTextMatcherNoTextMatcher
fieldMatcherTextMatcherNoTextMatcher
labelstringNo-
operationsDomOperation[]No-
parserstringNohtml
sourceCharsetstringNo-

Property Details​

contentTypeMatcher​

By default, this filter applies only to documents with content types specified by the CommonRestrictions.domContentTypes(String) method. You can also define your own content types if they represent files containing HTML or XML-like markup tags.

Common content-types:​

  • application/atom+xml
  • application/mathml+xml
  • application/rss+xml
  • application/vnd.wap.xhtml+xml
  • application/x-asp
  • application/xhtml+xml
  • application/xml
  • application/xslt+xml
  • image/svg+xml
  • text/html
  • text/xml

fieldMatcher​

Matches document fields based on specified patterns and methods

label​

An optional, user-supplied label for this step in the importer pipeline. It has no effect on processing — it exists purely to help identify this step, for example in large configurations or in the visual configurator.

operations​

DomOperation is used to manipulate or extract specific parts from the DOM document by selector. For example, one DomOperation extract TEXT from title and the other DomOperation extract HTML from div.content.

parser​

Defines using XML or HTML parser.

sourceCharset​

Sets the character encoding of the source document to ensure proper reading and processing.