GcsFetcher

Google Cloud Storage fetcher (gs://).

crawler/fsJavadoc: GcsFetcher

Notes

Fetches documents from Google Cloud Storage using references such as gs://my-bucket/path/to/object.txt.

This fetcher relies on the official google-cloud-nio provider.

Authentication is delegated to Google Application Default Credentials (ADC). When endpoint is set, the fetcher assumes a local/emulated GCS-compatible service and connects without authentication.

Examples

The following example uses default Google Application Default Credentials.

YAML

class: GcsFetcher

JSON

{
  "class": "GcsFetcher"
}

XML

<class>GcsFetcher</class>

The following example targets a local GCS-compatible endpoint.

YAML

class: GcsFetcher
endpoint: http://localhost:4443

JSON

{
  "class": "GcsFetcher",
  "endpoint": "http://localhost:4443"
}

XML

<class>GcsFetcher</class>
<endpoint>http://localhost:4443</endpoint>

Usage

Full configuration skeleton, with every option and its default

YAML

class: GcsFetcher
endpoint: string
referenceFilters: []

JSON

{
  "class": "GcsFetcher",
  "endpoint": "string",
  "referenceFilters": []
}

XML

<gcsFetcher>
  <class>GcsFetcher</class>
  <endpoint>string</endpoint>
  <referenceFilters>
    <referenceFilter/>
  </referenceFilters>
</gcsFetcher>

Properties

Property	Type	Required	Default
endpoint	string	No	-
referenceFilters	ReferenceFilter[]	No	-

Property Details

endpoint

Optional endpoint override, mainly intended for local GCS-compatible test servers.

referenceFilters

Filters applied to document references (URLs or file paths) to control which ones are fetched. Documents whose references do not pass the filters are skipped.