GenericUrlScopeResolver

Generic implementation that should satisfy most common site adherence scenarios.

crawler/webJavadoc: GenericUrlScopeResolver

Notes

Determines whether a discovered URL falls within the scope of the crawl by controlling domain, subdomain, port, and protocol adherence relative to the start URLs. Optionally restricts crawling to URLs listed in a detected sitemap.

Usage

YAML

class: GenericUrlScopeResolver
includeSubdomains: false
stayOnDomain: false
stayOnPort: false
stayOnProtocol: false
stayOnSitemap: false

JSON

{
  "class": "GenericUrlScopeResolver",
  "includeSubdomains": false,
  "stayOnDomain": false,
  "stayOnPort": false,
  "stayOnProtocol": false,
  "stayOnSitemap": false
}

XML

<urlScopeResolver>
  <class>GenericUrlScopeResolver</class>
  <includeSubdomains>false</includeSubdomains>
  <stayOnDomain>false</stayOnDomain>
  <stayOnPort>false</stayOnPort>
  <stayOnProtocol>false</stayOnProtocol>
  <stayOnSitemap>false</stayOnSitemap>
</urlScopeResolver>

Properties

Property	Type	Required	Default
includeSubdomains	boolean	No	false
stayOnDomain	boolean	No	false
stayOnPort	boolean	No	false
stayOnProtocol	boolean	No	false
stayOnSitemap	boolean	No	false

Property Details

includeSubdomains

Whether sub-domains are considered to be the same as a start URL domain.

stayOnDomain

Whether the crawler should always stay on the same domain name as the domain of each start URLs. By default the crawler will try follow any discovered links not otherwise rejected by other configuration settings (like regular filtering rules you may have).

stayOnPort

Whether the crawler should always stay on the same port as the port for each URL specified as a start URL. By default the crawler will try follow any discovered links not otherwise rejected by other settings (like regular filtering rules you may have).

stayOnProtocol

Whether the crawler should always stay on the same protocol as the protocol for each URL specified as a start URL. By default the crawler will try follow any discovered links not otherwise rejected by other settings (like regular filtering rules you may have).

stayOnSitemap

If a sitemap is provided or detected for a site, only crawl URLs that are defined in that sitemap.