Package com.norconex.crawler.web.util

Class Web

java.lang.Object

com.norconex.crawler.web.util.Web

public final class Web
extends Object

Method Summary

Modifier and Type	Method	Description
static WebCrawlerConfig	config(CrawlerContext crawlContext)
static void	fireIfUrlOutOfScope(CrawlerSession crawlSession, WebCrawlerEntry docContext, UrlScope urlScope)
static <T> CacheMap<T>	gridCache(@NonNull CrawlerSession crawlSession, @NonNull String name, @NonNull Class<? extends T> type)
static Properties	parseDomAttributes(String attribsStr)	Parses a string containing one or more HTML/XML attributes (e.g., key="value") and returns them all.
static Properties	parseDomAttributes(String attribsStr, boolean caseInsensitive)	Parses a string containing one or more HTML/XML attributes (e.g., key="value") and returns them all.
static RobotsTxt	robotsTxt(CrawlerContext crawler, String reference)
static String	trimAfterSubString(String wholeStr, String subStr)
static String	trimAroundSubString(String wholeStr, String subStr)
static String	trimBeforeSubString(String wholeStr, String subStr)

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Method Details

gridCache

public static <T> CacheMap<T> gridCache(@NonNull
 @NonNull CrawlerSession crawlSession,
 @NonNull
 @NonNull String name,
 @NonNull
 @NonNull Class<? extends T> type)

fireIfUrlOutOfScope

public static void fireIfUrlOutOfScope(CrawlerSession crawlSession,
 WebCrawlerEntry docContext,
 UrlScope urlScope)

config

public static WebCrawlerConfig config(CrawlerContext crawlContext)

robotsTxt

public static RobotsTxt robotsTxt(CrawlerContext crawler,
 String reference)

trimAroundSubString

public static String trimAroundSubString(String wholeStr,
 String subStr)

trimBeforeSubString

public static String trimBeforeSubString(String wholeStr,
 String subStr)

trimAfterSubString

public static String trimAfterSubString(String wholeStr,
 String subStr)

parseDomAttributes

public static Properties parseDomAttributes(String attribsStr)

Parses a string containing one or more HTML/XML attributes (e.g., key="value") and returns them all. This method deals with attributes only but can also be passed actual HTML/XML mark-up. Given the latter, this method will only consider attributes of the first element encountered. No attempt is made to first create a DOM model, so the string argument does not have to be fully "valid" XML/HTML. If the attribute string is null or produces no match, an empty Properties is returned.

Parameters:

attribsStr - the string containing attributes

Returns:

attributes (never null)

parseDomAttributes

public static Properties parseDomAttributes(String attribsStr,
 boolean caseInsensitive)

Parses a string containing one or more HTML/XML attributes (e.g., key="value") and returns them all. This method deals with attributes only but can also be passed actual HTML/XML mark-up. Given the latter, this method will only consider attributes of the first element encountered. No attempt is made to first create a DOM model, so the string argument does not have to be fully "valid" XML/HTML. If the attribute string is null or produces no match, an empty Properties is returned.

Parameters:

attribsStr - the string containing attributes

caseInsensitive - whether the return properties has case-insensitive keys

Returns:

attributes (never null)