AzureBlobFetcher

Azure Blob Storage fetcher (azblob:// and azureblob://).

crawler/fsJavadoc: AzureBlobFetcher

Notes

Fetches documents from Azure Blob Storage using references such as azblob://myaccount/mycontainer/path/to/blob.txt.

Supports:

• Shared-key authentication via credentials
• SAS token authentication via sasToken
• Optional endpoint override via endpoint

When endpoint is not set, the endpoint is derived from the reference account as https://<account>.blob.core.windows.net.

The credentials username must match the account in the reference URI.

Examples

The following example uses shared-key authentication.

YAML

class: AzureBlobFetcher
credentials:
  username: myaccount
  password: my-account-key

JSON

{
  "class": "AzureBlobFetcher",
  "credentials": {
    "username": "myaccount",
    "password": "my-account-key"
  }
}

XML

<class>AzureBlobFetcher</class>
<credentials>
  <username>myaccount</username>
  <password>my-account-key</password>
</credentials>

The following example uses a SAS token.

YAML

class: AzureBlobFetcher
sasToken: sv=2025-01-05&ss=bfqt&srt=sco&sp=rl&se=2027-01-01T00:00:00Z&sig=...

JSON

{
  "class": "AzureBlobFetcher",
  "sasToken": "sv=2025-01-05&ss=bfqt&srt=sco&sp=rl&se=2027-01-01T00:00:00Z&sig=..."
}

XML

<class>AzureBlobFetcher</class>
<sasToken>sv=2025-01-05&amp;ss=bfqt&amp;srt=sco&amp;sp=rl&amp;se=2027-01-01T00:00:00Z&amp;sig=...</sasToken>

Usage

Full configuration skeleton, with every option and its default

YAML

class: AzureBlobFetcher
credentials:
  password: string
  passwordKey:
    size: 0
    source: KEY
    value: string
  username: string
domain: string
endpoint: string
referenceFilters: []
sasToken: string

JSON

{
  "class": "AzureBlobFetcher",
  "credentials": {
    "password": "string",
    "passwordKey": {
      "size": 0,
      "source": "KEY",
      "value": "string"
    },
    "username": "string"
  },
  "domain": "string",
  "endpoint": "string",
  "referenceFilters": [],
  "sasToken": "string"
}

XML

<azureBlobFetcher>
  <class>AzureBlobFetcher</class>
  <credentials>
    <password>string</password>
    <passwordKey>
      <size>0</size>
      <source>KEY</source>
      <value>string</value>
    </passwordKey>
    <username>string</username>
  </credentials>
  <domain>string</domain>
  <endpoint>string</endpoint>
  <referenceFilters>
    <referenceFilter/>
  </referenceFilters>
  <sasToken>string</sasToken>
</azureBlobFetcher>

Properties

Property	Type	Required	Default
credentials	Credentials	No	Credentials
domain	string	No	-
endpoint	string	No	-
referenceFilters	ReferenceFilter[]	No	-
sasToken	string	No	-

Property Details

credentials

Username and password used for Azure Blob shared-key authentication:

• Username: storage account name
• Password: storage account key

domain

The domain name for authentication when accessing the remote resource.

endpoint

Optional service endpoint override. When omitted, the endpoint is derived from the reference account.

referenceFilters

Filters applied to document references (URLs or file paths) to control which ones are fetched. Documents whose references do not pass the filters are skipped.

sasToken

Optional SAS token used instead of shared-key credentials.