HdfsFetcher

Apache Hadoop File System (HDFS) fetcher.

crawler/fsJavadoc: HdfsFetcher

Notes

Fetches documents from Apache Hadoop File System (HDFS) using the webhdfs:// scheme. Supports simple authentication and Kerberos/SPNEGO authentication.

Usage

YAML

class: HdfsFetcher
authMethod: SIMPLE
credentials:
  password: string
  passwordKey:
    size: 0
    source: KEY
    value: string
  username: string
domain: string
kerberosConfig:
  keytabPath: string
  krb5ConfigPath: string
  loginModuleName: string
  principal: string
  servicePrincipalName: string
  useTicketCache: false
referenceFilters: []

JSON

{
  "class": "HdfsFetcher",
  "authMethod": "SIMPLE",
  "credentials": {
    "password": "string",
    "passwordKey": {
      "size": 0,
      "source": "KEY",
      "value": "string"
    },
    "username": "string"
  },
  "domain": "string",
  "kerberosConfig": {
    "keytabPath": "string",
    "krb5ConfigPath": "string",
    "loginModuleName": "string",
    "principal": "string",
    "servicePrincipalName": "string",
    "useTicketCache": false
  },
  "referenceFilters": []
}

XML

<hdfsFetcher>
  <class>HdfsFetcher</class>
  <authMethod>SIMPLE</authMethod>
  <credentials>
    <password>string</password>
    <passwordKey>
      <size>0</size>
      <source>KEY</source>
      <value>string</value>
    </passwordKey>
    <username>string</username>
  </credentials>
  <domain>string</domain>
  <kerberosConfig>
    <keytabPath>string</keytabPath>
    <krb5ConfigPath>string</krb5ConfigPath>
    <loginModuleName>string</loginModuleName>
    <principal>string</principal>
    <servicePrincipalName>string</servicePrincipalName>
    <useTicketCache>false</useTicketCache>
  </kerberosConfig>
  <referenceFilters>
    <referenceFilter/>
  </referenceFilters>
</hdfsFetcher>

Properties

Property	Type	Required	Default
authMethod	enum	No	SIMPLE
credentials	Credentials	No	Credentials
domain	string	No	-
kerberosConfig	KerberosConfig	No	-
referenceFilters	ReferenceFilter[]	No	-

Property Details

authMethod

Authentication method. Defaults to SIMPLE. Set to KERBEROS to use Kerberos/SPNEGO authentication.

Allowed Values

• SIMPLE
• KERBEROS

credentials

Optional credentials for the target HDFS. With SIMPLE, the username is sent as the user.name query parameter. With KERBEROS, credentials can be used for username/password-based JAAS login.

domain

The domain name for authentication when accessing the remote resource.

kerberosConfig

Kerberos/SPNEGO settings used when authMethod is KERBEROS. See KerberosConfig for details.

referenceFilters

Filters applied to document references (URLs or file paths) to control which ones are fetched. Documents whose references do not pass the filters are skipped.