GenericUrlNormalizer

Generic implementation that should satisfy most URL normalization needs.

crawler/webJavadoc: GenericUrlNormalizer

Notes

Normalizes crawled URLs to reduce duplicate content caused by equivalent URLs that differ only in format. Supports built-in normalization rules (such as lowercasing the scheme and host, removing default ports, and encoding non-URI characters) and custom search-and-replace patterns.

Examples

The following adds a normalization to add www. to URL domains when missing, to the default set of normalizations. It also add custom URL "search-and-replace" to remove any &view=print strings from URLs as well as replace &type=summary with &type=full.

YAML

class: GenericUrlNormalizer
normalizations:
  - REMOVE_FRAGMENT
  - LOWERCASE_SCHEME_HOST
  - UPPERCASE_ESCAPESEQUENCE
  - DECODE_UNRESERVED_CHARACTERS
  - REMOVE_DEFAULT_PORT
  - ENCODE_NON_URI_CHARACTERS
  - ADD_WWW
replacements:
  - match: "&view=print"
    value: ""
  - match: (&type=)(summary)
    value: $1full

JSON

{
  "class": "GenericUrlNormalizer",
  "normalizations": [
    "REMOVE_FRAGMENT",
    "LOWERCASE_SCHEME_HOST",
    "UPPERCASE_ESCAPESEQUENCE",
    "DECODE_UNRESERVED_CHARACTERS",
    "REMOVE_DEFAULT_PORT",
    "ENCODE_NON_URI_CHARACTERS",
    "ADD_WWW"
  ],
  "replacements": [
    {
      "match": "&view=print",
      "value": ""
    },
    {
      "match": "(&type=)(summary)",
      "value": "$1full"
    }
  ]
}

XML

<class>GenericUrlNormalizer</class>
<normalizations>
  <normalization>REMOVE_FRAGMENT</normalization>
  <normalization>LOWERCASE_SCHEME_HOST</normalization>
  <normalization>UPPERCASE_ESCAPESEQUENCE</normalization>
  <normalization>DECODE_UNRESERVED_CHARACTERS</normalization>
  <normalization>REMOVE_DEFAULT_PORT</normalization>
  <normalization>ENCODE_NON_URI_CHARACTERS</normalization>
  <normalization>ADD_WWW</normalization>
</normalizations>
<replacements>
  <replacement>
    <match>&amp;amp;view=print</match>
    <value/>
  </replacement>
  <replacement>
    <match>(&amp;amp;type=)(summary)</match>
    <value>$1full</value>
  </replacement>
</replacements>

Usage

Full configuration skeleton, with every option and its default

YAML

class: GenericUrlNormalizer
normalizations:
  - ADD_DIRECTORY_TRAILING_SLASH
replacements:
  - match: string
    value: string

JSON

{
  "class": "GenericUrlNormalizer",
  "normalizations": [
    "ADD_DIRECTORY_TRAILING_SLASH"
  ],
  "replacements": [
    {
      "match": "string",
      "value": "string"
    }
  ]
}

XML

<genericUrlNormalizer>
  <class>GenericUrlNormalizer</class>
  <normalizations>
    <normalization>ADD_DIRECTORY_TRAILING_SLASH</normalization>
  </normalizations>
  <replacements>
    <replacement>
      <match>string</match>
      <value>string</value>
    </replacement>
  </replacements>
</genericUrlNormalizer>

Properties

Property	Type	Required	Default
normalizations	enum[]	No	[REMOVE_FRAGMENT, LOWERCASE_SCHEME_HOST, UPPERCASE_ESCAPESEQUENCE, DECODE_UNRESERVED_CHARACTERS, REMOVE_DEFAULT_PORT, ENCODE_NON_URI_CHARACTERS]
replacements	NormalizationReplace[]	No	-

Property Details

normalizations

URL normalization rules. Rules prefixed with an asterisk (*) preserve semantic equivalence, as per the RFC 3986 standard.

replacements

Replace one or more matching portions of URLs with different text.