S3Fetcher

Amazon S3 fetcher (s3://).

crawler/fsJavadoc: S3Fetcher

Notes

Fetches documents from Amazon S3 using references such as s3://my-bucket/path/to/object.txt.

Supports AWS default credentials, explicit access keys via credentials, and S3-compatible endpoints.

The fetcher is designed to support common S3-compatible object stores (for example MinIO, Ceph RGW, and Cloudflare R2) by combining endpoint, region, and forcePathStyle as needed by the target platform.

S3 has no real directories. Folder-like paths are derived from object key prefixes.

Examples

The following example uses AWS default credentials and region resolution.

YAML

class: S3Fetcher

JSON

{
  "class": "S3Fetcher"
}

XML

<class>S3Fetcher</class>

The following example targets a MinIO endpoint.

YAML

class: S3Fetcher
endpoint: http://localhost:9000
forcePathStyle: true
credentials:
  username: minioadmin
  password: minioadmin

JSON

{
  "class": "S3Fetcher",
  "endpoint": "http://localhost:9000",
  "forcePathStyle": true,
  "credentials": {
    "username": "minioadmin",
    "password": "minioadmin"
  }
}

XML

<class>S3Fetcher</class>
<endpoint>http://localhost:9000</endpoint>
<forcePathStyle>true</forcePathStyle>
<credentials>
  <username>minioadmin</username>
  <password>minioadmin</password>
</credentials>

The following example targets a Ceph RGW endpoint.

YAML

class: S3Fetcher
endpoint: https://ceph.example.com
region: us-east-1
forcePathStyle: true
credentials:
  username: access-key
  password: secret-key

JSON

{
  "class": "S3Fetcher",
  "endpoint": "https://ceph.example.com",
  "region": "us-east-1",
  "forcePathStyle": true,
  "credentials": {
    "username": "access-key",
    "password": "secret-key"
  }
}

XML

<class>S3Fetcher</class>
<endpoint>https://ceph.example.com</endpoint>
<region>us-east-1</region>
<forcePathStyle>true</forcePathStyle>
<credentials>
  <username>access-key</username>
  <password>secret-key</password>
</credentials>

The following example targets a Cloudflare R2 endpoint.

YAML

class: S3Fetcher
endpoint: https://<account-id>.r2.cloudflarestorage.com
region: auto
credentials:
  username: r2-access-key
  password: r2-secret-key

JSON

{
  "class": "S3Fetcher",
  "endpoint": "https://<account-id>.r2.cloudflarestorage.com",
  "region": "auto",
  "credentials": {
    "username": "r2-access-key",
    "password": "r2-secret-key"
  }
}

XML

<class>S3Fetcher</class>
<endpoint>https://&lt;account-id&gt;.r2.cloudflarestorage.com</endpoint>
<region>auto</region>
<credentials>
  <username>r2-access-key</username>
  <password>r2-secret-key</password>
</credentials>

Usage

Full configuration skeleton, with every option and its default

YAML

class: S3Fetcher
credentials:
  password: string
  passwordKey:
    size: 0
    source: KEY
    value: string
  username: string
domain: string
endpoint: string
forcePathStyle: false
referenceFilters: []
region: string

JSON

{
  "class": "S3Fetcher",
  "credentials": {
    "password": "string",
    "passwordKey": {
      "size": 0,
      "source": "KEY",
      "value": "string"
    },
    "username": "string"
  },
  "domain": "string",
  "endpoint": "string",
  "forcePathStyle": false,
  "referenceFilters": [],
  "region": "string"
}

XML

<s3Fetcher>
  <class>S3Fetcher</class>
  <credentials>
    <password>string</password>
    <passwordKey>
      <size>0</size>
      <source>KEY</source>
      <value>string</value>
    </passwordKey>
    <username>string</username>
  </credentials>
  <domain>string</domain>
  <endpoint>string</endpoint>
  <forcePathStyle>false</forcePathStyle>
  <referenceFilters>
    <referenceFilter/>
  </referenceFilters>
  <region>string</region>
</s3Fetcher>

Properties

Property	Type	Required	Default
credentials	Credentials	No	Credentials
domain	string	No	-
endpoint	string	No	-
forcePathStyle	boolean	No	false
referenceFilters	ReferenceFilter[]	No	-
region	string	No	-

Property Details

credentials

Optional AWS credentials for authentication.

• Username: access key ID
• Password: secret access key

If not set, the AWS SDK default credential provider chain is used.

domain

The domain name for authentication when accessing the remote resource.

endpoint

Optional custom endpoint URL for S3-compatible services (for example MinIO).

forcePathStyle

Whether path-style bucket addressing is used (endpoint/bucket/key) instead of virtual-hosted style (bucket.endpoint/key).

referenceFilters

Filters applied to document references (URLs or file paths) to control which ones are fetched. Documents whose references do not pass the filters are skipped.

region

AWS region where the bucket lives (for example us-east-1). If not set, the AWS SDK default region provider chain is used.