HazelcastClusterConnector

Connects distributed crawler nodes using Hazelcast.

crawler/coreJavadoc: HazelcastClusterConnector

Notes

Connects crawler nodes using Hazelcast for distributed crawling. Supports cluster naming, instance naming, node expiry timeout, and worker heartbeat configuration.

Usage

YAML

class: HazelcastClusterConnector
clusterName: string
configurer:
  class: JdbcHazelcastConfigurer
  autoDiscoveryEnabled: false
  backupCount: 0
  columnKeyType: string
  columnValueType: string
  hazelcastProperties: {}
  initialLoadMode: LAZY
  jdbcDriver: string
  jdbcPassword: string
  jdbcUrl: string
  jdbcUsername: string
  jetEnabled: false
  maxPoolSize: 0
  queueStoreEnabled: false
  sqlMerge: string
  tcpMembers: string
  writeBatchSize: 0
  writeCoalescing: false
  writeDelaySeconds: 0
instanceName: string
nodeExpiryTimeout: 0
workerHeartbeatInterval: 0

JSON

{
  "class": "HazelcastClusterConnector",
  "clusterName": "string",
  "configurer": {
    "class": "JdbcHazelcastConfigurer",
    "autoDiscoveryEnabled": false,
    "backupCount": 0,
    "columnKeyType": "string",
    "columnValueType": "string",
    "hazelcastProperties": {},
    "initialLoadMode": "LAZY",
    "jdbcDriver": "string",
    "jdbcPassword": "string",
    "jdbcUrl": "string",
    "jdbcUsername": "string",
    "jetEnabled": false,
    "maxPoolSize": 0,
    "queueStoreEnabled": false,
    "sqlMerge": "string",
    "tcpMembers": "string",
    "writeBatchSize": 0,
    "writeCoalescing": false,
    "writeDelaySeconds": 0
  },
  "instanceName": "string",
  "nodeExpiryTimeout": 0,
  "workerHeartbeatInterval": 0
}

XML

<connector>
  <class>HazelcastClusterConnector</class>
  <clusterName>string</clusterName>
  <configurer>
    <class>JdbcHazelcastConfigurer</class>
    <autoDiscoveryEnabled>false</autoDiscoveryEnabled>
    <backupCount>0</backupCount>
    <columnKeyType>string</columnKeyType>
    <columnValueType>string</columnValueType>
    <hazelcastProperties/>
    <initialLoadMode>LAZY</initialLoadMode>
    <jdbcDriver>string</jdbcDriver>
    <jdbcPassword>string</jdbcPassword>
    <jdbcUrl>string</jdbcUrl>
    <jdbcUsername>string</jdbcUsername>
    <jetEnabled>false</jetEnabled>
    <maxPoolSize>0</maxPoolSize>
    <queueStoreEnabled>false</queueStoreEnabled>
    <sqlMerge>string</sqlMerge>
    <tcpMembers>string</tcpMembers>
    <writeBatchSize>0</writeBatchSize>
    <writeCoalescing>false</writeCoalescing>
    <writeDelaySeconds>0</writeDelaySeconds>
  </configurer>
  <instanceName>string</instanceName>
  <nodeExpiryTimeout>0</nodeExpiryTimeout>
  <workerHeartbeatInterval>0</workerHeartbeatInterval>
</connector>

Properties

Property	Type	Required	Default
clusterName	string	No	crawler-cluster
configurer	HazelcastConfigurer	No	JdbcHazelcastConfigurer
instanceName	string	No	-
nodeExpiryTimeout	integer	No	30000
workerHeartbeatInterval	integer	No	1000

Property Details

clusterName

The name identifying the Hazelcast cluster. All nodes sharing the same cluster name will automatically join the same cluster. Default is "crawler-cluster".

configurer

The configurer responsible for building the Hazelcast configuration. Defaults to JdbcHazelcastConfigurer, which uses JDBC persistence with an embedded H2 database in standalone mode. Advanced users can provide a custom HazelcastConfigurer implementation.

instanceName

Name of the Hazelcast instance to use for this cluster connector.

nodeExpiryTimeout

The maximum time to wait before declaring a node as expired in a distributed crawl. Minimum effective value is 5 seconds. Default is 30 seconds. Not applicable in standalone mode.

workerHeartbeatInterval

How frequently (in milliseconds) a worker signals that it is still alive. Used to detect stalled or crashed workers.