ArchiveFetcher

Fetches documents contained within archive files (ZIP, TAR, etc.).

crawler/fsJavadoc: ArchiveFetcher

Notes

Fetches documents from archive files (such as ZIP or TAR) on the local file system. Supports authentication settings for archive-backed remote file systems (for example, FTP or SFTP) and reference-level filtering.

Usage

YAML

class: ArchiveFetcher
credentials:
  password: string
  passwordKey:
    size: 0
    source: KEY
    value: string
  username: string
domain: string
referenceFilters: []

JSON

{
  "class": "ArchiveFetcher",
  "credentials": {
    "password": "string",
    "passwordKey": {
      "size": 0,
      "source": "KEY",
      "value": "string"
    },
    "username": "string"
  },
  "domain": "string",
  "referenceFilters": []
}

XML

<archiveFetcher>
  <class>ArchiveFetcher</class>
  <credentials>
    <password>string</password>
    <passwordKey>
      <size>0</size>
      <source>KEY</source>
      <value>string</value>
    </passwordKey>
    <username>string</username>
  </credentials>
  <domain>string</domain>
  <referenceFilters>
    <referenceFilter/>
  </referenceFilters>
</archiveFetcher>

Properties

Property	Type	Required	Default
credentials	Credentials	No	Credentials
domain	string	No	-
referenceFilters	ReferenceFilter[]	No	-

Property Details

credentials

Username and password used to authenticate with archive-backed remote file systems (for example, FTP or SFTP).

domain

The domain name for authentication when accessing the remote resource.

referenceFilters

Filters applied to document references (URLs or file paths) to control which ones are fetched. Documents whose references do not pass the filters are skipped.