Package com.norconex.crawler.web.doc.operations.scope.impl

Class GenericUrlScopeResolverConfig

java.lang.Object
com.norconex.crawler.web.doc.operations.scope.impl.GenericUrlScopeResolverConfig
public class GenericUrlScopeResolverConfig extends Object

Configuration for GenericUrlScopeResolver.

Since:
2.3.0

  • Constructor Details

    • GenericUrlScopeResolverConfig

      public GenericUrlScopeResolverConfig()

  • Method Details

    • isStayOnDomain

      public boolean isStayOnDomain()
      Whether the crawler should always stay on the same domain name as the domain for each URL specified as a start URL. By default (false) the crawler will try follow any discovered links not otherwise rejected by other settings (like regular filtering rules you may have).

    • isIncludeSubdomains

      public boolean isIncludeSubdomains()
      Whether sub-domains are considered to be the same as a URL domain. Only applicable when "stayOnDomain" is true.
      Since:
      2.9.0

    • isStayOnPort

      public boolean isStayOnPort()
      Whether the crawler should always stay on the same port as the port for each URL specified as a start URL. By default (false) the crawler will try follow any discovered links not otherwise rejected by other settings (like regular filtering rules you may have).

    • isStayOnProtocol

      public boolean isStayOnProtocol()
      Whether the crawler should always stay on the same protocol as the protocol for each URL specified as a start URL. By default (false) the crawler will try follow any discovered links not otherwise rejected by other settings (like regular filtering rules you may have).

    • isStayOnSitemap

      public boolean isStayOnSitemap()
      Whether to limit crawling to entries found in an existing website sitemap (do not go deeper). Only applies if a sitemap is present for a website. This option is similar to specifying a sitemap start URL with a maxDepth of 1 with the difference that when used with regular start URLs and no sitemap is detected, it will crawl the corresponding website as if this option was set to false. Does not apply if sitemap resolution has been disabled in your configuration. Note that if async is true, you may get a few soft rejections until they are identified in the sitemap.

    • setStayOnDomain

      public GenericUrlScopeResolverConfig setStayOnDomain(boolean stayOnDomain)
      Whether the crawler should always stay on the same domain name as the domain for each URL specified as a start URL. By default (false) the crawler will try follow any discovered links not otherwise rejected by other settings (like regular filtering rules you may have).
      Returns:
      this.

    • setIncludeSubdomains

      public GenericUrlScopeResolverConfig setIncludeSubdomains(boolean includeSubdomains)
      Whether sub-domains are considered to be the same as a URL domain. Only applicable when "stayOnDomain" is true.
      Returns:
      this.
      Since:
      2.9.0

    • setStayOnPort

      public GenericUrlScopeResolverConfig setStayOnPort(boolean stayOnPort)
      Whether the crawler should always stay on the same port as the port for each URL specified as a start URL. By default (false) the crawler will try follow any discovered links not otherwise rejected by other settings (like regular filtering rules you may have).
      Returns:
      this.

    • setStayOnProtocol

      public GenericUrlScopeResolverConfig setStayOnProtocol(boolean stayOnProtocol)
      Whether the crawler should always stay on the same protocol as the protocol for each URL specified as a start URL. By default (false) the crawler will try follow any discovered links not otherwise rejected by other settings (like regular filtering rules you may have).
      Returns:
      this.

    • setStayOnSitemap

      public GenericUrlScopeResolverConfig setStayOnSitemap(boolean stayOnSitemap)
      Whether to limit crawling to entries found in an existing website sitemap (do not go deeper). Only applies if a sitemap is present for a website. This option is similar to specifying a sitemap start URL with a maxDepth of 1 with the difference that when used with regular start URLs and no sitemap is detected, it will crawl the corresponding website as if this option was set to false. Does not apply if sitemap resolution has been disabled in your configuration. Note that if async is true, you may get a few soft rejections until they are identified in the sitemap.
      Returns:
      this.

    • equals

      public boolean equals(Object o)
      Overrides:
      equals in class Object

    • canEqual

      protected boolean canEqual(Object other)

    • hashCode

      public int hashCode()
      Overrides:
      hashCode in class Object

    • toString

      public String toString()
      Overrides:
      toString in class Object