Skip to main content

DefaultParser

Generic document parser for raw content passthrough or text decoding

importerJavadoc: DefaultParser

Notes​

It uses Apache Tika for most of its supported content types. For unknown content types, it falls back to Tika's generic media detector/parser.

Built-in baseline, not a hard limit

The format coverage described for DefaultParser reflects built-in importer capabilities in Norconex Crawler. It is a practical baseline, not a fixed ceiling. Teams can extend parser behavior and pipeline logic for customer-specific formats and extraction requirements.

Four optional features hang off this parser, each with its own settings page: embedded for documents inside documents, ocr for text inside images, grobid for scientific papers, and sentiment for sentiment analysis.

Examples​

Saves documents that fail to parse into a directory for inspection.

handler:
  class: DefaultParser
  errorsSaveDir: C:\reports\errors

Uses Tesseract to read English text out of the images inside PDFs.

handler:
  class: DefaultParser
  ocr:
    tesseractPath: /app/ocr/tesseract
    contentTypeMatcher:
      method: BASIC
      pattern: application/pdf
      ignoreCase: true
    language: eng

Splits the contents of ZIP files into individual documents, skips images embedded in any container, and leaves the contents of Word documents alone while still processing the Word document itself.

handler:
  class: DefaultParser
  embedded:
    splitContentTypes:
      - method: BASIC
        pattern: application/zip
        ignoreCase: true
    skipEmbeddedContentTypes:
      - method: REGEX
        pattern: image/.*
        ignoreCase: true
    skipEmbeddedOfContentTypes:
      - method: BASIC
        pattern: application/msword
        ignoreCase: true

Usage​

Full configuration skeleton, with every option and its default
class: DefaultParser
embedded:
  maxEmbeddedDepth: 0
  skipEmbeddedContentTypes:
    - {}
  skipEmbeddedOfContentTypes:
    - {}
  splitContentTypes:
    - {}
errorsSaveDir: string
grobid:
  enabled: false
  serviceUrl: string
label: string
ocr:
  applyRotation: false
  colorSpace: string
  contentTypeMatcher:
    ignoreCase: false
    ignoreDiacritic: false
    matchEmpty: false
    method: BASIC
    negateMatches: false
    partial: false
    pattern: string
    replaceAll: false
    trim: false
  density: 0
  depth: 0
  disabled: false
  enableImagePreprocessing: false
  filter: string
  imageMagickPath: string
  language: string
  maxFileSizeToOcr: 0
  minFileSizeToOcr: 0
  pageSegMode: string
  pageSeparator: string
  preserveInterwordSpacing: false
  resize: 0
  skipOcr: false
  tessdataPath: string
  tesseractPath: string
  timeoutSeconds: 0
sentiment:
  enabled: false
  modelPath: string

Properties​

PropertyTypeRequiredDefault
embeddedEmbeddedConfigNoEmbeddedConfig
errorsSaveDirstringNo-
grobidGrobidConfigNoGrobidConfig
labelstringNo-
ocrOcrConfigNoOcrConfig
sentimentSentimentConfigNoSentimentConfig

Property Details​

embedded​

How documents contained inside other documents are handled — the files in a zip, the attachments on an email, the images in a PDF. By default their content and metadata are merged into the containing document.

Set splitContentTypes to have a container's embedded documents become documents in their own right, each going through the whole import cycle. Set skipEmbeddedContentTypes or skipEmbeddedOfContentTypes to leave some of them unparsed. See EmbeddedConfig.

errorsSaveDir​

Directory where documents that fail to parse are copied, for inspection. Left unset, failures are logged but the document is not kept.

grobid​

Optional Grobid service settings, for extracting structured metadata and text from scientific documents such as journal article PDFs. Disabled unless you enable it. See GrobidConfig.

label​

An optional, user-supplied label for this step in the importer pipeline. It has no effect on processing — it exists purely to help identify this step, for example in large configurations or in the visual configurator.

ocr​

Optional Tesseract settings, for pulling text out of images and of documents containing images — scanned PDFs, TIFF archives, screenshots. Tesseract is not bundled: install it separately and point ocr.tesseractPath at it, and OCR stays inactive until you do.

Once active, OCR is attempted on every supported image format (TIFF, PNG, JPEG, GIF, BMP, and images embedded in documents). It is slow next to text extraction, so restricting it by content type and file size is usually worth doing. All settings are described under OcrConfig.

sentiment​

Optional Tika sentiment analysis settings, for scoring document content as positive or negative. Disabled by default: unlike most Tika parsers, this one downloads its model from the network the moment it initializes, regardless of whether it is ever used, so leaving it off avoids startup delays or failures when that network call cannot complete. See SentimentConfig.