MVStoreClusterConnectorConfig

Storage tuning for the MVStore cluster connector.

crawler/coreJavadoc: MVStoreClusterConnectorConfig

Notes

Controls how the embedded MVStore database writes crawl state to disk. Every setting has a working default, and a crawl runs correctly without touching any of them — reach for these only when profiling points at the store itself, on crawls large enough for the difference to be measurable.

The trade-offs pull against each other: larger buffers and caches spend memory to reduce I/O, compression spends CPU to reduce disk, and more aggressive compaction spends both to keep the file small. Change one at a time.

Examples

YAML

  clusterConnector:
    class: MVStoreClusterConnector
    configuration:
      cacheSize: 64
      compress: 1

JSON

{
  "clusterConnector": {
    "class": "MVStoreClusterConnector",
    "configuration": {
      "cacheSize": 64,
      "compress": 1
    }
  }
}

XML

<clusterConnector class="MVStoreClusterConnector">
  <cacheSize>64</cacheSize>
  <compress>1</compress>
</clusterConnector>

Usage

Full configuration skeleton, with every option and its default

YAML

autoCommitBufferSize: 0
autoCommitDelay: 0
autoCompactFillRate: 0
cacheSize: 0
compress: 0
pageSplitSize: 0

JSON

{
  "autoCommitBufferSize": 0,
  "autoCommitDelay": 0,
  "autoCompactFillRate": 0,
  "cacheSize": 0,
  "compress": 0,
  "pageSplitSize": 0
}

XML

<mVStoreClusterConnectorConfig>
  <autoCommitBufferSize>0</autoCommitBufferSize>
  <autoCommitDelay>0</autoCommitDelay>
  <autoCompactFillRate>0</autoCompactFillRate>
  <cacheSize>0</cacheSize>
  <compress>0</compress>
  <pageSplitSize>0</pageSplitSize>
</mVStoreClusterConnectorConfig>

Properties

Property	Type	Required	Default
autoCommitBufferSize	integer	No	1024
autoCommitDelay	integer	No	1000
autoCompactFillRate	integer	No	40
cacheSize	integer	No	16
compress	integer	No	0
pageSplitSize	integer	No	4096

Property Details

autoCommitBufferSize

Size in KB of the write buffer before changes are committed automatically. Default is 1024 (1 MB). A larger buffer means fewer, bigger writes.

autoCommitDelay

How long in milliseconds before buffered changes are committed regardless of buffer size. Default is 1000 (1 second). Set to 0 to commit on buffer size alone.

autoCompactFillRate

Fill rate percentage (0–100) below which the store compacts itself. Default is 40. A lower value compacts less often, leaving a larger file; a higher value compacts more often, spending I/O to keep it small.

cacheSize

Read cache size in MB. Default is 16. Worth raising on crawls whose state no longer fits comfortably in the default cache.

compress

Compression level: 0 for none (the default), 1 for fast (LZF), 2 for high (Deflate). Compression trades CPU for disk space, and mostly pays off when document metadata is large and repetitive.

pageSplitSize

Page split size in bytes. Default is 4096. Larger pages can read long values faster in sequence, at the cost of wasted space for small entries.