Package com.norconex.crawler.web.doc.operations.url

Interface WebUrlNormalizer

All Known Implementing Classes:
GenericUrlNormalizer
public interface WebUrlNormalizer
Responsible for normalizing URLs. Normalization is taking a raw URL and modifying it to its most basic or standard form. In other words, this makes different URLs "equivalent". This allows to eliminate URL variations that points to the same content (e.g. URL carrying temporary session information). This action takes place right after URLs are extracted from a document, before each of these URLs is even considered for further processing. Returning null will effectively tell the crawler to not even consider it for processing (it won't go through the regular document processing flow). You may want to consider ReferenceFilter to exclude URLs as part has the regular document processing flow (may create a trace in the logs and gives you more options). Implementors also implementing XMLConfigurable must name their XML tag urlNormalizer to ensure it gets loaded properly.

  • Method Details

    • normalizeURL

      String normalizeURL(String url)
      Normalize the given URL.
      Parameters:
      url - the URL to normalize
      Returns:
      the normalized URL

    • normalizeURL

      static String normalizeURL(String url, List<WebUrlNormalizer> normalizers)
      Normalizes a URL by applying each normalizers in the list.
      Parameters:
      url - the URL to normalize
      normalizers - the normalizers
      Returns:
      the normalized URL