MVStoreClusterConnector

File-backed crawl state using H2's MVStore engine. The default connector.

crawler/coreJavadoc: MVStoreClusterConnector

Notes

Keeps crawl state in a local MVStore file, which is what makes a crawl resumable and incremental without any external infrastructure. This is the default: a crawler that declares no cluster connector runs on this one.

The cluster instance is cached, so several crawl sessions pointing at the same connector share the same file rather than opening it twice.

Being file-backed makes it single-node by definition. To spread a crawl across machines, use HazelcastClusterConnector; to keep everything in memory and discard it at exit, see MemoryClusterConnector. Storage tuning lives in MVStoreClusterConnectorConfig, and its defaults are fine for most crawls.

Examples

YAML

  clusterConnector:
    class: MVStoreClusterConnector

JSON

{
  "clusterConnector": {
    "class": "MVStoreClusterConnector"
  }
}

XML

<clusterConnector class="MVStoreClusterConnector"/>

Usage

Full configuration skeleton, with every option and its default

YAML

class: MVStoreClusterConnector
autoCommitBufferSize: 0
autoCommitDelay: 0
autoCompactFillRate: 0
cacheSize: 0
compress: 0
pageSplitSize: 0

JSON

{
  "class": "MVStoreClusterConnector",
  "autoCommitBufferSize": 0,
  "autoCommitDelay": 0,
  "autoCompactFillRate": 0,
  "cacheSize": 0,
  "compress": 0,
  "pageSplitSize": 0
}

XML

<connector>
  <class>MVStoreClusterConnector</class>
  <autoCommitBufferSize>0</autoCommitBufferSize>
  <autoCommitDelay>0</autoCommitDelay>
  <autoCompactFillRate>0</autoCompactFillRate>
  <cacheSize>0</cacheSize>
  <compress>0</compress>
  <pageSplitSize>0</pageSplitSize>
</connector>

Properties

Property	Type	Required	Default
autoCommitBufferSize	integer	No	1024
autoCommitDelay	integer	No	1000
autoCompactFillRate	integer	No	40
cacheSize	integer	No	16
compress	integer	No	0
pageSplitSize	integer	No	4096

Property Details

autoCommitBufferSize

Size in KB of the write buffer before changes are committed automatically. Default is 1024 (1 MB).

autoCommitDelay

How long in milliseconds before buffered changes are committed regardless of buffer size. Default is 1000. Set to 0 to commit on buffer size alone.

autoCompactFillRate

Fill rate percentage (0–100) below which the store compacts itself. Default is 40. Lower compacts less often and leaves a larger file.

cacheSize

Read cache size in MB. Default is 16.

compress

Compression level: 0 for none (the default), 1 for fast (LZF), 2 for high (Deflate).

pageSplitSize

Page split size in bytes. Default is 4096. Larger pages read long values faster in sequence, at the cost of wasted space for small entries.