Package com.norconex.crawler.web.doc.operations.url.impl

Class GenericUrlNormalizer

java.lang.Object
com.norconex.crawler.web.doc.operations.url.impl.GenericUrlNormalizer
All Implemented Interfaces:
Configurable<GenericUrlNormalizerConfig>, WebUrlNormalizer
public class GenericUrlNormalizer extends Object implements WebUrlNormalizer, Configurable<GenericUrlNormalizerConfig>

Generic implementation of WebUrlNormalizer that should satisfy most URL normalization needs. This implementation relies on UrlNormalizer. Please refer to it for complete documentation and examples.

This class is in effect by default. To skip its usage, you can explicitly set the URL Normalizer to null in the WebCrawlerConfig.

By default, this class removes the URL fragment and applies these RFC 3986 normalizations:

  • Converting the scheme and host to lower case
  • Capitalizing letters in escape sequences
  • Decoding percent-encoded unreserved characters
  • Removing the default port
  • Encoding non-URI characters

To overwrite this default, you have to specify a new list of normalizations to apply, via the GenericUrlNormalizerConfig.setNormalizations(List) method, or via XML configuration. Each normalizations is identified by a code name. The following is the complete code name list for supported normalizations. Click on any code name to get a full description from WebUrlNormalizer:

In addition, this class allows you to specify any number of URL value replacements using regular expressions.