MemoryClusterConnector

Crawl state held in memory only, with nothing written to disk.

crawler/coreJavadoc: MemoryClusterConnector

Notes

Keeps the whole crawl state in memory, with no file and no external infrastructure. Fast to start, nothing to clean up, and nothing survives the JVM exiting — the next run starts from scratch.

That makes it a good fit for tests, one-off crawls, and embedded runs where a state file would be in the way. It is the wrong fit for anything you intend to resume or crawl incrementally across runs; use MVStoreClusterConnector, the default, for that.

Within a single JVM the cluster is cached, so several crawl sessions using this connector do share state — incremental crawling works across sessions, just not across restarts.

Examples

YAML

  clusterConnector:
    class: MemoryClusterConnector

JSON

{
  "clusterConnector": {
    "class": "MemoryClusterConnector"
  }
}

XML

<clusterConnector class="MemoryClusterConnector"/>

Usage

Full configuration skeleton, with every option and its default

YAML

class: MemoryClusterConnector
noop: string

JSON

{
  "class": "MemoryClusterConnector",
  "noop": "string"
}

XML

<connector>
  <class>MemoryClusterConnector</class>
  <noop>string</noop>
</connector>