KerberosConfig

Kerberos/SPNEGO authentication settings for HDFS WebHDFS access.

crawler/webJavadoc: KerberosConfig

Notes

Shared by two fetchers: HdfsFetcher when its authMethod is KERBEROS, and the web HttpClientFetcher when its authentication method is KERBEROS or SPNEGO. servicePrincipalName applies to the web case only.

At a minimum, provide a valid Kerberos configuration (krb5.conf) and one of the following authentication mechanisms:

• keytab-based login (keytabPath + principal)
• ticket-cache login (useTicketCache)
• username/password JAAS login through the fetcher's credentials

When loginModuleName is not set, the crawler creates a built-in JAAS configuration from the provided properties.

Examples

Uses a keytab and principal for non-interactive authentication against HDFS.

YAML

class: HdfsFetcher
authMethod: KERBEROS
kerberosConfig:
krb5ConfigPath: /etc/krb5.conf
keytabPath: /etc/security/keytabs/hdfs-user.keytab
principal: hdfs-user@EXAMPLE.COM

JSON

{
  "class": "HdfsFetcher",
  "authMethod": "KERBEROS",
  "kerberosConfig": {
    "krb5ConfigPath": "/etc/krb5.conf",
    "keytabPath": "/etc/security/keytabs/hdfs-user.keytab",
    "principal": "hdfs-user@EXAMPLE.COM"
  }
}

XML

<fetcher class="HdfsFetcher">
  <authMethod>KERBEROS</authMethod>
  <kerberosConfig>
    <krb5ConfigPath>/etc/krb5.conf</krb5ConfigPath>
    <keytabPath>/etc/security/keytabs/hdfs-user.keytab</keytabPath>
    <principal>hdfs-user@EXAMPLE.COM</principal>
  </kerberosConfig>
</fetcher>

Uses whatever ticket is already in the system cache, as left by kinit. No keytab or password is needed, but the ticket has to be kept alive for the length of the crawl.

YAML

class: HdfsFetcher
authMethod: KERBEROS
kerberosConfig:
krb5ConfigPath: /etc/krb5.conf
useTicketCache: true

JSON

{
  "class": "HdfsFetcher",
  "authMethod": "KERBEROS",
  "kerberosConfig": {
    "krb5ConfigPath": "/etc/krb5.conf",
    "useTicketCache": true
  }
}

XML

<fetcher class="HdfsFetcher">
  <authMethod>KERBEROS</authMethod>
  <kerberosConfig>
    <krb5ConfigPath>/etc/krb5.conf</krb5ConfigPath>
    <useTicketCache>true</useTicketCache>
  </kerberosConfig>
</fetcher>

Web crawling against a SPNEGO-protected site, where the service principal identifies the target server.

YAML

authMethod: SPNEGO
kerberosConfig:
krb5ConfigPath: /etc/krb5.conf
servicePrincipalName: HTTP/intranet.example.com@EXAMPLE.COM
useTicketCache: true

JSON

{
  "authMethod": "SPNEGO",
  "kerberosConfig": {
    "krb5ConfigPath": "/etc/krb5.conf",
    "servicePrincipalName": "HTTP/intranet.example.com@EXAMPLE.COM",
    "useTicketCache": true
  }
}

XML

<authMethod>SPNEGO</authMethod>
<kerberosConfig>
  <krb5ConfigPath>/etc/krb5.conf</krb5ConfigPath>
  <servicePrincipalName>HTTP/intranet.example.com@EXAMPLE.COM</servicePrincipalName>
  <useTicketCache>true</useTicketCache>
</kerberosConfig>

Usage

Full configuration skeleton, with every option and its default

YAML

keytabPath: string
krb5ConfigPath: string
loginModuleName: string
principal: string
servicePrincipalName: string
useTicketCache: false

JSON

{
  "keytabPath": "string",
  "krb5ConfigPath": "string",
  "loginModuleName": "string",
  "principal": "string",
  "servicePrincipalName": "string",
  "useTicketCache": false
}

XML

<kerberosConfig>
  <keytabPath>string</keytabPath>
  <krb5ConfigPath>string</krb5ConfigPath>
  <loginModuleName>string</loginModuleName>
  <principal>string</principal>
  <servicePrincipalName>string</servicePrincipalName>
  <useTicketCache>false</useTicketCache>
</kerberosConfig>

Properties

Property	Type	Required	Default
keytabPath	string	No	-
krb5ConfigPath	string	No	-
loginModuleName	string	No	-
principal	string	No	-
servicePrincipalName	string	No	-
useTicketCache	boolean	No	false

Property Details

keytabPath

Path to the keytab file containing the principal key. When set, authentication uses this keytab instead of requiring a password or ticket cache.

krb5ConfigPath

Path to the Kerberos configuration file (krb5.conf). When omitted, the JVM default Kerberos configuration is used.

loginModuleName

Optional JAAS login module entry name. When omitted, an internal JAAS configuration is generated automatically.

principal

Kerberos principal name (for example, user@REALM). Required for keytab-based authentication.

servicePrincipalName

Kerberos service principal name (SPN) of the target service, typically in the form HTTP/hostname@REALM or HTTP/hostname.

useTicketCache

Whether the system ticket cache should be used. When enabled, an existing ticket (for example from kinit) is used and no keytab or password is required.