WebDavFetcher

Web Distributed Authoring and Versioning (WebDAV) fetcher.

crawler/fsJavadoc: WebDavFetcher

Notes

Fetches documents from WebDAV servers using the Web Distributed Authoring and Versioning protocol. Supports credentials, SSL/TLS configuration, connection pooling, redirect following, keep-alive, preemptive authentication, and proxy settings.

Usage

YAML

class: WebDavFetcher
connectionTimeout: 0
credentials:
  password: string
  passwordKey:
    size: 0
    source: KEY
    value: string
  username: string
domain: string
followRedirect: false
hostnameVerificationEnabled: false
keepAlive: false
keyStoreFile: string
keyStorePass: string
keyStorePassKey:
  size: 0
  source: KEY
  value: string
keyStoreType: string
maxConnectionsPerHost: 0
maxTotalConnections: 0
preemptiveAuth: false
proxyDomain: string
proxySettings:
  credentials:
    password: string
    passwordKey: {}
    username: string
  host:
    name: string
    port: 0
  realm: string
  scheme: string
referenceFilters: []
soTimeout: 0
tlsVersions: string
urlCharset: string
userAgent: string

JSON

{
  "class": "WebDavFetcher",
  "connectionTimeout": 0,
  "credentials": {
    "password": "string",
    "passwordKey": {
      "size": 0,
      "source": "KEY",
      "value": "string"
    },
    "username": "string"
  },
  "domain": "string",
  "followRedirect": false,
  "hostnameVerificationEnabled": false,
  "keepAlive": false,
  "keyStoreFile": "string",
  "keyStorePass": "string",
  "keyStorePassKey": {
    "size": 0,
    "source": "KEY",
    "value": "string"
  },
  "keyStoreType": "string",
  "maxConnectionsPerHost": 0,
  "maxTotalConnections": 0,
  "preemptiveAuth": false,
  "proxyDomain": "string",
  "proxySettings": {
    "credentials": {
      "password": "string",
      "passwordKey": {},
      "username": "string"
    },
    "host": {
      "name": "string",
      "port": 0
    },
    "realm": "string",
    "scheme": "string"
  },
  "referenceFilters": [],
  "soTimeout": 0,
  "tlsVersions": "string",
  "urlCharset": "string",
  "userAgent": "string"
}

XML

<webDavFetcher>
  <class>WebDavFetcher</class>
  <connectionTimeout>0</connectionTimeout>
  <credentials>
    <password>string</password>
    <passwordKey>
      <size>0</size>
      <source>KEY</source>
      <value>string</value>
    </passwordKey>
    <username>string</username>
  </credentials>
  <domain>string</domain>
  <followRedirect>false</followRedirect>
  <hostnameVerificationEnabled>false</hostnameVerificationEnabled>
  <keepAlive>false</keepAlive>
  <keyStoreFile>string</keyStoreFile>
  <keyStorePass>string</keyStorePass>
  <keyStorePassKey>
    <size>0</size>
    <source>KEY</source>
    <value>string</value>
  </keyStorePassKey>
  <keyStoreType>string</keyStoreType>
  <maxConnectionsPerHost>0</maxConnectionsPerHost>
  <maxTotalConnections>0</maxTotalConnections>
  <preemptiveAuth>false</preemptiveAuth>
  <proxyDomain>string</proxyDomain>
  <proxySettings>
    <credentials>
      <password>string</password>
      <passwordKey/>
      <username>string</username>
    </credentials>
    <host>
      <name>string</name>
      <port>0</port>
    </host>
    <realm>string</realm>
    <scheme>string</scheme>
  </proxySettings>
  <referenceFilters>
    <referenceFilter/>
  </referenceFilters>
  <soTimeout>0</soTimeout>
  <tlsVersions>string</tlsVersions>
  <urlCharset>string</urlCharset>
  <userAgent>string</userAgent>
</webDavFetcher>

Properties

Property	Type	Required	Default
connectionTimeout	integer	No	-
credentials	Credentials	No	Credentials
domain	string	No	-
followRedirect	boolean	No	false
hostnameVerificationEnabled	boolean	No	false
keepAlive	boolean	No	false
keyStoreFile	string	No	-
keyStorePass	string	No	-
keyStorePassKey	EncryptionKey	No	-
keyStoreType	string	No	-
maxConnectionsPerHost	integer	No	5
maxTotalConnections	integer	No	50
preemptiveAuth	boolean	No	false
proxyDomain	string	No	-
proxySettings	ProxySettings	No	ProxySettings
referenceFilters	ReferenceFilter[]	No	-
soTimeout	integer	No	-
tlsVersions	string	No	-
urlCharset	string	No	-
userAgent	string	No	-

Property Details

connectionTimeout

Maximum wait time for a requested connection to be fully established, in milliseconds. Leave blank for unlimited.

credentials

Username and password used to authenticate with the WebDAV endpoint.

domain

The domain name for authentication when accessing the remote resource.

followRedirect

Follow HTTP redirects for the connection.

hostnameVerificationEnabled

Verify the host name in SSL context.

keepAlive

Reuse HTTP connections.

keyStoreFile

Path to the keystore file for SSL connections.

keyStorePass

Keystore passphrase for SSL connections.

keyStorePassKey

Keystore passkey for SSL connections.

keyStoreType

Keystore type for SSL connections (e.g., dks, pkcs11, pkcs12, ...).

maxConnectionsPerHost

The maximum number of connections allowed to any host.

maxTotalConnections

The total maximum number of connections allowed.

preemptiveAuth

Whether to perform preemptive authentication (valid for basic authentication method).

proxyDomain

If required to authenticate to a proxy, the user's network domain.

proxySettings

Configures an optional HTTP proxy for outbound connections. Leave unset to connect directly.

referenceFilters

Filters applied to document references (URLs or file paths) to control which ones are fetched. Documents whose references do not pass the filters are skipped.

soTimeout

Maximum period of inactivity between two consecutive data packets.

tlsVersions

The enabled TLS versions as a comma-separated string (e.g., V_1_2, V_1_3).

urlCharset

Charset used for url encoding.

userAgent

The user agent to attach to the outgoing HTTP methods.