FtpFetcher

File Transfer Protocol (FTP and FTPS) fetcher.

crawler/fsJavadoc: FtpFetcher

Notes

Fetches documents from FTP and FTPS servers. Supports passive mode, explicit and implicit FTPS security modes, data channel protection, keep-alive settings, custom date formats, server encoding, and authentication configuration.

Examples

The following example sets FTP options that some hosts require to list files in server directories.

YAML

class: FtpFetcher
passiveMode: true
userDirIsRoot: false

JSON

{
  "class": "FtpFetcher",
  "passiveMode": true,
  "userDirIsRoot": false
}

XML

<class>FtpFetcher</class>
<passiveMode>true</passiveMode>
<userDirIsRoot>false</userDirIsRoot>

Usage

Full configuration skeleton, with every option and its default

YAML

class: FtpFetcher
autodetectUtf8: false
connectTimeout: 0
controlEncoding: string
controlKeepAliveReplyTimeout: 0
controlKeepAliveTimeout: 0
credentials:
  password: string
  passwordKey:
    size: 0
    source: KEY
    value: string
  username: string
dataChannelProtectionLevel: CLEAR
dataTimeout: 0
defaultDateFormat: string
domain: string
passiveMode: false
proxySettings:
  credentials:
    password: string
    passwordKey: {}
    username: string
  host:
    name: string
    port: 0
  realm: string
  scheme: string
recentDateFormat: string
referenceFilters: []
remoteVerificationDisabled: false
securityMode: IMPLICIT
serverLanguageCode: string
shortMonthNames:
  - string
socketTimeout: 0
userDirIsRoot: false

JSON

{
  "class": "FtpFetcher",
  "autodetectUtf8": false,
  "connectTimeout": 0,
  "controlEncoding": "string",
  "controlKeepAliveReplyTimeout": 0,
  "controlKeepAliveTimeout": 0,
  "credentials": {
    "password": "string",
    "passwordKey": {
      "size": 0,
      "source": "KEY",
      "value": "string"
    },
    "username": "string"
  },
  "dataChannelProtectionLevel": "CLEAR",
  "dataTimeout": 0,
  "defaultDateFormat": "string",
  "domain": "string",
  "passiveMode": false,
  "proxySettings": {
    "credentials": {
      "password": "string",
      "passwordKey": {},
      "username": "string"
    },
    "host": {
      "name": "string",
      "port": 0
    },
    "realm": "string",
    "scheme": "string"
  },
  "recentDateFormat": "string",
  "referenceFilters": [],
  "remoteVerificationDisabled": false,
  "securityMode": "IMPLICIT",
  "serverLanguageCode": "string",
  "shortMonthNames": [
    "string"
  ],
  "socketTimeout": 0,
  "userDirIsRoot": false
}

XML

<ftpFetcher>
  <class>FtpFetcher</class>
  <autodetectUtf8>false</autodetectUtf8>
  <connectTimeout>0</connectTimeout>
  <controlEncoding>string</controlEncoding>
  <controlKeepAliveReplyTimeout>0</controlKeepAliveReplyTimeout>
  <controlKeepAliveTimeout>0</controlKeepAliveTimeout>
  <credentials>
    <password>string</password>
    <passwordKey>
      <size>0</size>
      <source>KEY</source>
      <value>string</value>
    </passwordKey>
    <username>string</username>
  </credentials>
  <dataChannelProtectionLevel>CLEAR</dataChannelProtectionLevel>
  <dataTimeout>0</dataTimeout>
  <defaultDateFormat>string</defaultDateFormat>
  <domain>string</domain>
  <passiveMode>false</passiveMode>
  <proxySettings>
    <credentials>
      <password>string</password>
      <passwordKey/>
      <username>string</username>
    </credentials>
    <host>
      <name>string</name>
      <port>0</port>
    </host>
    <realm>string</realm>
    <scheme>string</scheme>
  </proxySettings>
  <recentDateFormat>string</recentDateFormat>
  <referenceFilters>
    <referenceFilter/>
  </referenceFilters>
  <remoteVerificationDisabled>false</remoteVerificationDisabled>
  <securityMode>IMPLICIT</securityMode>
  <serverLanguageCode>string</serverLanguageCode>
  <shortMonthNames>
    <shortMonthName>string</shortMonthName>
  </shortMonthNames>
  <socketTimeout>0</socketTimeout>
  <userDirIsRoot>false</userDirIsRoot>
</ftpFetcher>

Properties

Property	Type	Required	Default
autodetectUtf8	boolean	No	false
connectTimeout	integer	No	-
controlEncoding	string	No	-
controlKeepAliveReplyTimeout	integer	No	-
controlKeepAliveTimeout	integer	No	-
credentials	Credentials	No	Credentials
dataChannelProtectionLevel	enum	No	-
dataTimeout	integer	No	-
defaultDateFormat	string	No	-
domain	string	No	-
passiveMode	boolean	No	false
proxySettings	ProxySettings	No	ProxySettings
recentDateFormat	string	No	-
referenceFilters	ReferenceFilter[]	No	-
remoteVerificationDisabled	boolean	No	false
securityMode	enum	No	-
serverLanguageCode	string	No	-
shortMonthNames	string[]	No	-
socketTimeout	integer	No	-
userDirIsRoot	boolean	No	false

Property Details

autodetectUtf8

Autodetect if the server is using UTF-8.

connectTimeout

The timeout for the initial control connection. Leave blank for no timeout.

controlEncoding

Save the character encoding to be used by the FTP control connection.

controlKeepAliveReplyTimeout

The control keep alive reply timeout for the FTP client.

controlKeepAliveTimeout

The control keep alive timeout for the FTP client.

credentials

Username and password used to authenticate with the FTP or FTPS server.

dataChannelProtectionLevel

The data channel protection level (PROT).

Allowed Values

• CLEAR
• SAFE
• CONFIDENTIAL
• PRIVATE

dataTimeout

The data timeout for the FTP client. Leave blank for no timeout.

defaultDateFormat

The default date format used by the server.

domain

The domain name for authentication when accessing the remote resource.

passiveMode

Use FTP PASV command to connect in passive mode.

proxySettings

Configures an optional HTTP proxy for outbound connections. Leave unset to connect directly.

recentDateFormat

The server date format, if different than US format.

referenceFilters

Filters applied to document references (URLs or file paths) to control which ones are fetched. Documents whose references do not pass the filters are skipped.

remoteVerificationDisabled

Disable verification that the remote host making a connection to the FTP client is the intended one (based on host and port).

securityMode

FTPS security mode.

Allowed Values

• IMPLICIT
• EXPLICIT

serverLanguageCode

The server language code, if different than English (en).

shortMonthNames

A set of month names used by the server if different than those expected by the client (considering the server language code).

socketTimeout

The maximum time in milliseconds to wait for data on an established connection before giving up.

userDirIsRoot

When enabled, the user's home directory on the remote server is treated as the root directory for navigation.