AdlsGen2Fetcher

Azure Data Lake Storage Gen2 fetcher (abfs:// and abfss://).

crawler/fsJavadoc: AdlsGen2Fetcher

Notes

Fetches documents from Azure Data Lake Storage Gen2 using references such as abfss://filesystem@account.dfs.core.windows.net/path/to/file.

Supports:

• Shared key authentication via credentials
• SAS token authentication via sasToken
• Optional endpoint override via endpoint
• Optional ACL/owner/group extraction control via aclDisabled

When endpoint is not set, the endpoint is derived from the reference account as https://<account>.dfs.core.windows.net.

The credentials username must match the account in the reference URI.

Examples

The following example uses shared-key authentication and crawls an ADLS Gen2 file system.

YAML

class: AdlsGen2Fetcher
credentials:
  username: myaccount
  password: my-account-key

JSON

{
  "class": "AdlsGen2Fetcher",
  "credentials": {
    "username": "myaccount",
    "password": "my-account-key"
  }
}

XML

<class>AdlsGen2Fetcher</class>
<credentials>
  <username>myaccount</username>
  <password>my-account-key</password>
</credentials>

The following example uses a SAS token.

YAML

class: AdlsGen2Fetcher
sasToken: sv=2025-01-05&ss=bfqt&srt=sco&sp=rl&se=2027-01-01T00:00:00Z&sig=...

JSON

{
  "class": "AdlsGen2Fetcher",
  "sasToken": "sv=2025-01-05&ss=bfqt&srt=sco&sp=rl&se=2027-01-01T00:00:00Z&sig=..."
}

XML

<class>AdlsGen2Fetcher</class>
<sasToken>sv=2025-01-05&amp;ss=bfqt&amp;srt=sco&amp;sp=rl&amp;se=2027-01-01T00:00:00Z&amp;sig=...</sasToken>

The following example disables ACL extraction.

YAML

class: AdlsGen2Fetcher
aclDisabled: true

JSON

{
  "class": "AdlsGen2Fetcher",
  "aclDisabled": true
}

XML

<class>AdlsGen2Fetcher</class>
<aclDisabled>true</aclDisabled>

Usage

Full configuration skeleton, with every option and its default

YAML

class: AdlsGen2Fetcher
aclDisabled: false
credentials:
  password: string
  passwordKey:
    size: 0
    source: KEY
    value: string
  username: string
domain: string
endpoint: string
referenceFilters: []
sasToken: string

JSON

{
  "class": "AdlsGen2Fetcher",
  "aclDisabled": false,
  "credentials": {
    "password": "string",
    "passwordKey": {
      "size": 0,
      "source": "KEY",
      "value": "string"
    },
    "username": "string"
  },
  "domain": "string",
  "endpoint": "string",
  "referenceFilters": [],
  "sasToken": "string"
}

XML

<adlsGen2Fetcher>
  <class>AdlsGen2Fetcher</class>
  <aclDisabled>false</aclDisabled>
  <credentials>
    <password>string</password>
    <passwordKey>
      <size>0</size>
      <source>KEY</source>
      <value>string</value>
    </passwordKey>
    <username>string</username>
  </credentials>
  <domain>string</domain>
  <endpoint>string</endpoint>
  <referenceFilters>
    <referenceFilter/>
  </referenceFilters>
  <sasToken>string</sasToken>
</adlsGen2Fetcher>

Properties

Property	Type	Required	Default
aclDisabled	boolean	No	false
credentials	Credentials	No	Credentials
domain	string	No	-
endpoint	string	No	-
referenceFilters	ReferenceFilter[]	No	-
sasToken	string	No	-

Property Details

aclDisabled

Whether ACL, owner, and group metadata extraction is disabled.

credentials

Username and password used for ADLS Gen2 shared-key authentication:

• Username: storage account name
• Password: storage account key

domain

The domain name for authentication when accessing the remote resource.

endpoint

Optional service endpoint override. When omitted, the endpoint is derived from the reference account.

referenceFilters

Filters applied to document references (URLs or file paths) to control which ones are fetched. Documents whose references do not pass the filters are skipped.

sasToken

Optional SAS token used instead of shared-key credentials.