Uses of Class
com.norconex.crawler.web.WebCrawlerConfig

Packages that use WebCrawlerConfig

Package	Description
com.norconex.crawler.web
com.norconex.crawler.web.util

Uses of WebCrawlerConfig in com.norconex.crawler.web

Methods in com.norconex.crawler.web that return WebCrawlerConfig

Modifier and Type	Method	Description
WebCrawlerConfig	WebCrawlerConfig.setCanonicalLinkDetector(CanonicalLinkDetector canonicalLinkDetector)	The canonical link detector.
WebCrawlerConfig	WebCrawlerConfig.setDelayResolver(DelayResolver delayResolver)	The delay resolver dictating the minimum amount of time to wait between web requests.
WebCrawlerConfig	WebCrawlerConfig.setKeepReferencedLinks(Set<WebCrawlerConfig.ReferencedLinkType> keepReferencedLinks)	Sets whether to keep referenced links and what to keep.
WebCrawlerConfig	WebCrawlerConfig.setLinkExtractors(List<LinkExtractor> linkExtractors)	Sets link extractors.
WebCrawlerConfig	WebCrawlerConfig.setPostImportLinks(TextMatcher fieldMatcher)	Set a field matcher used to identify post-import metadata fields holding URLs to consider for crawling.
WebCrawlerConfig	WebCrawlerConfig.setPostImportLinksKeep(boolean postImportLinksKeep)	Whether to keep the Importer-populated fields from getPostImportLinks().
WebCrawlerConfig	WebCrawlerConfig.setRecrawlableResolver(RecrawlableResolver recrawlableResolver)	The resolver that indicates whether a given URL is ready to be crawled by a new crawl session.
WebCrawlerConfig	WebCrawlerConfig.setRobotsMetaProvider(RobotsMetaProvider robotsMetaProvider)	The provider of robots metadata rules for a page (if applicable).
WebCrawlerConfig	WebCrawlerConfig.setRobotsTxtProvider(RobotsTxtProvider robotsTxtProvider)	The provider of robots.txt rules for a site (if applicable).
WebCrawlerConfig	WebCrawlerConfig.setSitemapLocator(SitemapLocator sitemapLocator)	The locator of sitemaps (if applicable).
WebCrawlerConfig	WebCrawlerConfig.setSitemapResolver(SitemapResolver sitemapResolver)	The resolver of web site sitemaps (if applicable).
WebCrawlerConfig	WebCrawlerConfig.setStartReferencesSitemaps(List<String> startReferencesSitemaps)	Sets the sitemap URLs used as starting points for crawling.
WebCrawlerConfig	WebCrawlerConfig.setUrlNormalizers(List<WebUrlNormalizer> urlNormalizers)	Sets URL normalizers.
WebCrawlerConfig	WebCrawlerConfig.setUrlScopeResolver(UrlScopeResolver urlScopeResolver)	The strategy to use to determine if a URL is in scope.

Methods in com.norconex.crawler.web with parameters of type WebCrawlerConfig

Modifier and Type	Method	Description
static Crawler	WebCrawler.create(WebCrawlerConfig crawlerConfig)	Creates a Web Crawler instance.

Uses of WebCrawlerConfig in com.norconex.crawler.web.util

Methods in com.norconex.crawler.web.util that return WebCrawlerConfig

Modifier and Type	Method	Description
static WebCrawlerConfig	Web.config(CrawlerContext crawlContext)