SaveDocumentTransformer

Saves document content or metadata to a specified directory.

importerJavadoc: SaveDocumentTransformer

Notes

Saves a copy of the document in its current processing state to a specified directory. To save the original, unmodified file, it is recommended to use this transformer as a pre-parse handler.

Examples

Saves a copy of each document under /path/to/save/documents, recording where it went in a file.path field.

YAML

handler:
  class: SaveDocumentTransformer
  saveDir: /path/to/save/documents
  pathToField: file.path
  defaultFileName: index.html

JSON

{
  "handler": {
    "class": "SaveDocumentTransformer",
    "saveDir": "/path/to/save/documents",
    "pathToField": "file.path",
    "defaultFileName": "index.html"
  }
}

XML

<handler>
  <class>SaveDocumentTransformer</class>
  <saveDir>/path/to/save/documents</saveDir>
  <pathToField>file.path</pathToField>
  <defaultFileName>index.html</defaultFileName>
</handler>

Usage

Full configuration skeleton, with every option and its default

YAML

class: SaveDocumentTransformer
defaultFileName: string
dirSplitPattern: string
escape: false
label: string
maxPathLength: 0
pathToField: string
saveDir: string

JSON

{
  "class": "SaveDocumentTransformer",
  "defaultFileName": "string",
  "dirSplitPattern": "string",
  "escape": false,
  "label": "string",
  "maxPathLength": 0,
  "pathToField": "string",
  "saveDir": "string"
}

XML

<handler>
  <class>SaveDocumentTransformer</class>
  <defaultFileName>string</defaultFileName>
  <dirSplitPattern>string</dirSplitPattern>
  <escape>false</escape>
  <label>string</label>
  <maxPathLength>0</maxPathLength>
  <pathToField>string</pathToField>
  <saveDir>string</saveDir>
</handler>

Properties

Property	Type	Required	Default
defaultFileName	string	No	default.file
dirSplitPattern	string	No	[\:/]
escape	boolean	No	false
label	string	No	-
maxPathLength	integer	No	-1
pathToField	string	No	-
saveDir	string	No	-

Property Details

defaultFileName

File name used when the path derived from the reference turns out to be a directory — for a reference ending in /, or one that is a prefix of another already saved. Defaults to default.file; index.html is a natural choice when saving crawled web pages.

dirSplitPattern

Regular expression matching the separators used to split a document reference into sub-directories. Defaults to [\:/], which turns https://example.com/a/b.html into https/example.com/a/b.html. Set it to a pattern that matches nothing to save every document in one flat directory.

escape

Whether each path segment is rewritten into a file-system-safe name. Default is false. Turn it on when references contain characters the file system rejects — query strings, colons, wildcards — otherwise saving them fails.

label

An optional, user-supplied label for this step in the importer pipeline. It has no effect on processing — it exists purely to help identify this step, for example in large configurations or in the visual configurator.

maxPathLength

Maximum length of the full saved path, including saveDir. Paths longer than this are truncated and given a short hash to keep them unique. Default is -1, and any value under 10 also means unlimited.

Useful on Windows, where the classic path limit is 260 characters and long crawled URLs overrun it. If saveDir itself is already too long to leave room for the hash, truncation is skipped and a warning is logged once.

pathToField

Name of a metadata field in which to store the path of the saved file. Leave it unset if you don't need the document to carry its own save location.

saveDir

Directory the copies are written to. Defaults to ./savedDocuments, relative to the working directory. Sub-directories are created under it following dirSplitPattern.