Package com.norconex.crawler.core.cluster

Class ClusterConfig

java.lang.Object
com.norconex.crawler.core.cluster.ClusterConfig
public class ClusterConfig extends Object

  • Field Details

    • DEFAULT_ADMIN_PORT

      public static final int DEFAULT_ADMIN_PORT
      Default port used to communicate via a node for some administrative crawler cluster tasks.
      See Also:

    • ADMIN_BIND_LOOPBACK

      public static final String ADMIN_BIND_LOOPBACK
      setAdminBindAddress(String) value binding the administrative server to the loopback interface only, reachable from the host running the crawler and nowhere else.
      See Also:

    • ADMIN_BIND_ANY

      public static final String ADMIN_BIND_ANY
      setAdminBindAddress(String) value binding the administrative server to every network interface, so other hosts can reach it.
      See Also:

    • DEFAULT_ADMIN_BIND_ADDRESS

      public static final String DEFAULT_ADMIN_BIND_ADDRESS
      Default administrative server bind address: loopback only.
      See Also:

  • Constructor Details

    • ClusterConfig

      public ClusterConfig()

  • Method Details

    • getConnector

      public ClusterConnector getConnector()
      The connector to the cluster implementation used to run the crawler. Default is an MVStore file-backed connector with no external infrastructure. For clustered mode, set this to a HazelcastClusterConnector. For a pure in-memory non-persistent alternative, use MemoryClusterConnector.

    • isAdminDisabled

      public boolean isAdminDisabled()
      Disable launching the crawler administrative server endpoints.

    • getAdminPort

      public int getAdminPort()
      Port the crawler cluster listens to for administrative commands, on each nodes. Incremented to the next available port in case of conflicts. Default is 27295 (mnemonic: ‘CRAWL’ on a phone keypad).

    • getAdminBindAddress

      public String getAdminBindAddress()

      Network interface the administrative server binds to. Accepted values:

      • "loopback" (default) — the loopback interface only. The administrative endpoints are then reachable from the host running the crawler and from nowhere else.
      • "any" — every interface. Required for clustered mode, where nodes and the stop command may run on other hosts.
      • any host name or IP address — that interface only.

      The default is deliberately the restrictive one. The administrative endpoints include stopping the crawl, and they are guarded only by a crawler-id request header, which is an identifier taken from the configuration rather than a secret. Exposing them beyond the host means anyone who can reach the port and name the crawler can halt a crawl, so widening this is an explicit decision.

    • isClustered

      public boolean isClustered()
      Whether the crawler should run in stand-alone or cluster mode. Default is standalone, optimized for performance, by eliminating or minimizing inter-node communication.

    • setConnector

      public ClusterConfig setConnector(ClusterConnector connector)
      The connector to the cluster implementation used to run the crawler. Default is an MVStore file-backed connector with no external infrastructure. For clustered mode, set this to a HazelcastClusterConnector. For a pure in-memory non-persistent alternative, use MemoryClusterConnector.
      Returns:
      this.

    • setAdminDisabled

      public ClusterConfig setAdminDisabled(boolean adminDisabled)
      Disable launching the crawler administrative server endpoints.
      Returns:
      this.

    • setAdminPort

      public ClusterConfig setAdminPort(int adminPort)
      Port the crawler cluster listens to for administrative commands, on each nodes. Incremented to the next available port in case of conflicts. Default is 27295 (mnemonic: ‘CRAWL’ on a phone keypad).
      Returns:
      this.

    • setAdminBindAddress

      public ClusterConfig setAdminBindAddress(String adminBindAddress)

      Network interface the administrative server binds to. Accepted values:

      • "loopback" (default) — the loopback interface only. The administrative endpoints are then reachable from the host running the crawler and from nowhere else.
      • "any" — every interface. Required for clustered mode, where nodes and the stop command may run on other hosts.
      • any host name or IP address — that interface only.

      The default is deliberately the restrictive one. The administrative endpoints include stopping the crawl, and they are guarded only by a crawler-id request header, which is an identifier taken from the configuration rather than a secret. Exposing them beyond the host means anyone who can reach the port and name the crawler can halt a crawl, so widening this is an explicit decision.

      Returns:
      this.

    • setClustered

      public ClusterConfig setClustered(boolean clustered)
      Whether the crawler should run in stand-alone or cluster mode. Default is standalone, optimized for performance, by eliminating or minimizing inter-node communication.
      Returns:
      this.

    • equals

      public boolean equals(Object o)
      Overrides:
      equals in class Object

    • canEqual

      protected boolean canEqual(Object other)

    • hashCode

      public int hashCode()
      Overrides:
      hashCode in class Object

    • toString

      public String toString()
      Overrides:
      toString in class Object