SftpFetcher

Secure File Transfer Protocol (SFTP) fetcher.

crawler/fsJavadoc: SftpFetcher

Notes

Fetches documents from SFTP (SSH File Transfer Protocol) servers. Supports credentials, compression, known hosts verification, preferred authentication method ordering, file name encoding, and strict host key checking.

Usage

YAML

class: SftpFetcher
compression: string
connectTimeout: 0
credentials:
  password: string
  passwordKey:
    size: 0
    source: KEY
    value: string
  username: string
domain: string
fileNameEncoding: string
knownHosts: string
preferredAuthentications: string
referenceFilters: []
strictHostKeyChecking: string
userDirIsRoot: false

JSON

{
  "class": "SftpFetcher",
  "compression": "string",
  "connectTimeout": 0,
  "credentials": {
    "password": "string",
    "passwordKey": {
      "size": 0,
      "source": "KEY",
      "value": "string"
    },
    "username": "string"
  },
  "domain": "string",
  "fileNameEncoding": "string",
  "knownHosts": "string",
  "preferredAuthentications": "string",
  "referenceFilters": [],
  "strictHostKeyChecking": "string",
  "userDirIsRoot": false
}

XML

<sftpFetcher>
  <class>SftpFetcher</class>
  <compression>string</compression>
  <connectTimeout>0</connectTimeout>
  <credentials>
    <password>string</password>
    <passwordKey>
      <size>0</size>
      <source>KEY</source>
      <value>string</value>
    </passwordKey>
    <username>string</username>
  </credentials>
  <domain>string</domain>
  <fileNameEncoding>string</fileNameEncoding>
  <knownHosts>string</knownHosts>
  <preferredAuthentications>string</preferredAuthentications>
  <referenceFilters>
    <referenceFilter/>
  </referenceFilters>
  <strictHostKeyChecking>string</strictHostKeyChecking>
  <userDirIsRoot>false</userDirIsRoot>
</sftpFetcher>

Properties

Property	Type	Required	Default
compression	string	No	-
connectTimeout	integer	No	-
credentials	Credentials	No	Credentials
domain	string	No	-
fileNameEncoding	string	No	-
knownHosts	string	No	-
preferredAuthentications	string	No	-
referenceFilters	ReferenceFilter[]	No	-
strictHostKeyChecking	string	No	no
userDirIsRoot	boolean	No	false

Property Details

compression

The compression algorithms to use to enable compression (e.g., "zlib,none").

connectTimeout

Timeout value to create the connection. Leave blank for no timeout.

credentials

Username and password used to authenticate with the SFTP server.

domain

The domain name for authentication when accessing the remote resource.

fileNameEncoding

The character encoding to use for file names.

knownHosts

Path to the known_hosts file (e.g., /home/user/.ssh/known_hosts2).

preferredAuthentications

Comma-separated order of authentication methods to try (e.g., publickey,password,keyboard-interactive).

referenceFilters

Filters applied to document references (URLs or file paths) to control which ones are fetched. Documents whose references do not pass the filters are skipped.

strictHostKeyChecking

Whether to verify the host key ("yes", "no" or "ask").

userDirIsRoot

When enabled, the user's home directory on the remote server is treated as the root directory for navigation.