Package com.norconex.crawler.core.cluster.impl.hazelcast

Class JdbcHazelcastConfigurer

java.lang.Object
com.norconex.crawler.core.cluster.impl.hazelcast.JdbcHazelcastConfigurer
All Implemented Interfaces:
HazelcastConfigurer
public class JdbcHazelcastConfigurer extends Object implements HazelcastConfigurer
Default HazelcastConfigurer implementation that builds the Hazelcast configuration programmatically using a JDBC data source for persistence.

Standalone mode (default, clustered = false): An embedded H2 file-based database stored in the crawler's work directory is used automatically — no external configuration is required.

Clustered mode (clustered = true): An external database must be supplied via setJdbcUrl(String) (plus optionally setJdbcUsername(String), setJdbcPassword(String) , setJdbcDriver(String)). The default SQL merge statement uses PostgreSQL INSERT ... ON CONFLICT syntax; set setSqlMerge(String) to target a different database dialect.

Effective defaults by mode
SettingStandaloneClustered
backupCount01
maxPoolSize1020
jdbcUrlH2 file in workDirmust be set
sqlMergeH2 MERGE syntax PostgreSQL INSERT … ON CONFLICT

  • Field Details

    • MAP_STORE_FACTORY_CLASS

      public static final String MAP_STORE_FACTORY_CLASS
      Factory class for JDBC-backed map stores.

    • QUEUE_STORE_FACTORY_CLASS

      public static final String QUEUE_STORE_FACTORY_CLASS
      Factory class for JDBC-backed queue stores.

    • DATA_CONNECTION_REF

      public static final String DATA_CONNECTION_REF
      Name of the shared JDBC data-connection referenced by store configs.
      See Also:

  • Constructor Details

    • JdbcHazelcastConfigurer

      public JdbcHazelcastConfigurer()

  • Method Details

    • buildConfig

      public com.hazelcast.config.Config buildConfig(HazelcastConfigurerContext ctx)
      Description copied from interface: HazelcastConfigurer
      Builds a Hazelcast Config for the given context. The returned config must not yet have a HazelcastInstance started against it.
      Specified by:
      buildConfig in interface HazelcastConfigurer
      Parameters:
      ctx - context carrying work directory, clustering mode, and cluster name
      Returns:
      a fully-configured (but not yet started) Hazelcast Config

    • getJdbcUrl

      public String getJdbcUrl()
      JDBC URL for the data source. In standalone mode, defaults to an H2 file database stored inside the crawler work directory. In clustered mode this must be set explicitly.

    • getJdbcUsername

      public String getJdbcUsername()
      JDBC username. Defaults to "sa" (H2 default).

    • getJdbcPassword

      public String getJdbcPassword()
      JDBC password. Defaults to empty string.

    • getJdbcDriver

      public String getJdbcDriver()
      JDBC driver class name. Optional; HikariCP auto-detects the driver from the URL when this is not set.

    • getMaxPoolSize

      public int getMaxPoolSize()
      Maximum JDBC connection pool size. 0 (default) means automatic: 10 for standalone, 20 for clustered.

    • getWriteDelaySeconds

      public int getWriteDelaySeconds()
      Write-behind delay in seconds for data maps (ledger, session, etc.). 0 makes writes synchronous (write-through). Default is 5, which batches writes for significantly better throughput. Pipeline coordination maps always use write-through regardless of this setting.

    • getWriteBatchSize

      public int getWriteBatchSize()
      Maximum number of entries per write-behind batch. Only effective when writeDelaySeconds > 0. Default is 100.

    • isWriteCoalescing

      public boolean isWriteCoalescing()
      Whether to coalesce write-behind updates so that only the latest value for each key is persisted per batch. Only effective when writeDelaySeconds > 0. Default is true.

    • getColumnKeyType

      public String getColumnKeyType()
      SQL column type for map/queue keys. Defaults to "VARCHAR(4096)".

    • getColumnValueType

      public String getColumnValueType()
      SQL column type for map values. Defaults to "TEXT".

    • getSqlMerge

      public String getSqlMerge()
      SQL merge/upsert template used for map stores. The token {tableName} is replaced at runtime with the actual table name.

      Defaults to H2 MERGE … KEY … syntax for standalone and PostgreSQL INSERT … ON CONFLICT … syntax for clustered. Override this when targeting a different database dialect.

    • getInitialLoadMode

      public com.hazelcast.config.MapStoreConfig.InitialLoadMode getInitialLoadMode()
      Hazelcast map-store initial load mode. MapStoreConfig.InitialLoadMode.LAZY (default) defers loading until an entry is accessed, which speeds up startup. Critical maps such as the crawl ledger call loadAll() explicitly when they are first used so that operations like size() and iteration return correct results. MapStoreConfig.InitialLoadMode.EAGER loads all entries as soon as the cluster member starts.

    • getBackupCount

      public int getBackupCount()
      Hazelcast backup count. -1 (default) means automatic: 0 for standalone, 1 for clustered.

    • getTcpMembers

      public String getTcpMembers()
      TCP/IP member list for clustered discovery, as a comma-separated list of host:port addresses (e.g., "192.168.1.10:5701,192.168.1.11:5701"). Defaults to "127.0.0.1:5701,127.0.0.1:5702,127.0.0.1:5703" when in clustered mode and not otherwise specified.

    • isAutoDiscoveryEnabled

      public boolean isAutoDiscoveryEnabled()
      Whether to use Hazelcast auto-discovery in clustered mode.

      Default is false to keep deterministic TCP member discovery. When set to true, auto-detection is enabled and explicit TCP-member discovery is disabled.

    • getJetEnabled

      public Boolean getJetEnabled()
      Whether Hazelcast Jet engine should be enabled.

      null (default) means enabled for backward compatibility. Set to false when Jet is not used and a leaner cluster runtime is preferred (e.g., deterministic integration tests).

    • isQueueStoreEnabled

      public boolean isQueueStoreEnabled()
      Whether queue-store persistence should be enabled for Hazelcast queues.

      Defaults to true. Set to false for tests that do not require persistent queue recovery and prioritize startup/runtime speed.

    • getHazelcastProperties

      public Map<String,String> getHazelcastProperties()
      Additional Hazelcast system properties set on the config (e.g. "hazelcast.partition.count" → "17"). Applied after all other settings are built.

    • setJdbcUrl

      public JdbcHazelcastConfigurer setJdbcUrl(String jdbcUrl)
      JDBC URL for the data source. In standalone mode, defaults to an H2 file database stored inside the crawler work directory. In clustered mode this must be set explicitly.
      Returns:
      this.

    • setJdbcUsername

      public JdbcHazelcastConfigurer setJdbcUsername(String jdbcUsername)
      JDBC username. Defaults to "sa" (H2 default).
      Returns:
      this.

    • setJdbcPassword

      public JdbcHazelcastConfigurer setJdbcPassword(String jdbcPassword)
      JDBC password. Defaults to empty string.
      Returns:
      this.

    • setJdbcDriver

      public JdbcHazelcastConfigurer setJdbcDriver(String jdbcDriver)
      JDBC driver class name. Optional; HikariCP auto-detects the driver from the URL when this is not set.
      Returns:
      this.

    • setMaxPoolSize

      public JdbcHazelcastConfigurer setMaxPoolSize(int maxPoolSize)
      Maximum JDBC connection pool size. 0 (default) means automatic: 10 for standalone, 20 for clustered.
      Returns:
      this.

    • setWriteDelaySeconds

      public JdbcHazelcastConfigurer setWriteDelaySeconds(int writeDelaySeconds)
      Write-behind delay in seconds for data maps (ledger, session, etc.). 0 makes writes synchronous (write-through). Default is 5, which batches writes for significantly better throughput. Pipeline coordination maps always use write-through regardless of this setting.
      Returns:
      this.

    • setWriteBatchSize

      public JdbcHazelcastConfigurer setWriteBatchSize(int writeBatchSize)
      Maximum number of entries per write-behind batch. Only effective when writeDelaySeconds > 0. Default is 100.
      Returns:
      this.

    • setWriteCoalescing

      public JdbcHazelcastConfigurer setWriteCoalescing(boolean writeCoalescing)
      Whether to coalesce write-behind updates so that only the latest value for each key is persisted per batch. Only effective when writeDelaySeconds > 0. Default is true.
      Returns:
      this.

    • setColumnKeyType

      public JdbcHazelcastConfigurer setColumnKeyType(String columnKeyType)
      SQL column type for map/queue keys. Defaults to "VARCHAR(4096)".
      Returns:
      this.

    • setColumnValueType

      public JdbcHazelcastConfigurer setColumnValueType(String columnValueType)
      SQL column type for map values. Defaults to "TEXT".
      Returns:
      this.

    • setSqlMerge

      public JdbcHazelcastConfigurer setSqlMerge(String sqlMerge)
      SQL merge/upsert template used for map stores. The token {tableName} is replaced at runtime with the actual table name.

      Defaults to H2 MERGE … KEY … syntax for standalone and PostgreSQL INSERT … ON CONFLICT … syntax for clustered. Override this when targeting a different database dialect.

      Returns:
      this.

    • setInitialLoadMode

      public JdbcHazelcastConfigurer setInitialLoadMode(com.hazelcast.config.MapStoreConfig.InitialLoadMode initialLoadMode)
      Hazelcast map-store initial load mode. MapStoreConfig.InitialLoadMode.LAZY (default) defers loading until an entry is accessed, which speeds up startup. Critical maps such as the crawl ledger call loadAll() explicitly when they are first used so that operations like size() and iteration return correct results. MapStoreConfig.InitialLoadMode.EAGER loads all entries as soon as the cluster member starts.
      Returns:
      this.

    • setBackupCount

      public JdbcHazelcastConfigurer setBackupCount(int backupCount)
      Hazelcast backup count. -1 (default) means automatic: 0 for standalone, 1 for clustered.
      Returns:
      this.

    • setTcpMembers

      public JdbcHazelcastConfigurer setTcpMembers(String tcpMembers)
      TCP/IP member list for clustered discovery, as a comma-separated list of host:port addresses (e.g., "192.168.1.10:5701,192.168.1.11:5701"). Defaults to "127.0.0.1:5701,127.0.0.1:5702,127.0.0.1:5703" when in clustered mode and not otherwise specified.
      Returns:
      this.

    • setAutoDiscoveryEnabled

      public JdbcHazelcastConfigurer setAutoDiscoveryEnabled(boolean autoDiscoveryEnabled)
      Whether to use Hazelcast auto-discovery in clustered mode.

      Default is false to keep deterministic TCP member discovery. When set to true, auto-detection is enabled and explicit TCP-member discovery is disabled.

      Returns:
      this.

    • setJetEnabled

      public JdbcHazelcastConfigurer setJetEnabled(Boolean jetEnabled)
      Whether Hazelcast Jet engine should be enabled.

      null (default) means enabled for backward compatibility. Set to false when Jet is not used and a leaner cluster runtime is preferred (e.g., deterministic integration tests).

      Returns:
      this.

    • setQueueStoreEnabled

      public JdbcHazelcastConfigurer setQueueStoreEnabled(boolean queueStoreEnabled)
      Whether queue-store persistence should be enabled for Hazelcast queues.

      Defaults to true. Set to false for tests that do not require persistent queue recovery and prioritize startup/runtime speed.

      Returns:
      this.

    • setHazelcastProperties

      public JdbcHazelcastConfigurer setHazelcastProperties(Map<String,String> hazelcastProperties)
      Additional Hazelcast system properties set on the config (e.g. "hazelcast.partition.count" → "17"). Applied after all other settings are built.
      Returns:
      this.

    • equals

      public boolean equals(Object o)
      Overrides:
      equals in class Object

    • canEqual

      protected boolean canEqual(Object other)

    • hashCode

      public int hashCode()
      Overrides:
      hashCode in class Object

    • toString

      public String toString()
      Overrides:
      toString in class Object