Normalization

Supported URL normalization operations.

crawler/webJavadoc: Normalization

Notes

Identifies a specific URL normalization operation to apply, such as removing default ports, lowercasing the scheme and host, or removing trailing slashes. Multiple normalization values can be combined to produce canonical URL forms.

Usage

YAML

Normalization: ADD_DIRECTORY_TRAILING_SLASH

JSON

{
  "Normalization": "ADD_DIRECTORY_TRAILING_SLASH"
}

XML

<normalization/>

Allowed Values

• ADD_DIRECTORY_TRAILING_SLASH
• ADD_DOMAIN_TRAILING_SLASH
• ADD_WWW
• DECODE_UNRESERVED_CHARACTERS
• ENCODE_NON_URI_CHARACTERS
• ENCODE_SPACES
• LOWERCASE
• LOWERCASE_PATH
• LOWERCASE_QUERY
• LOWERCASE_QUERY_PARAMETER_NAMES
• LOWERCASE_QUERY_PARAMETER_VALUES
• LOWERCASE_SCHEME_HOST
• REMOVE_DEFAULT_PORT
• REMOVE_DIRECTORY_INDEX
• REMOVE_DOT_SEGMENTS
• REMOVE_DUPLICATE_SLASHES
• REMOVE_EMPTY_PARAMETERS
• REMOVE_FRAGMENT
• REMOVE_QUERY_STRING
• REMOVE_SESSION_IDS
• REMOVE_TRAILING_FRAGMENT
• REMOVE_TRAILING_QUESTION_MARK
• REMOVE_TRAILING_SLASH
• REMOVE_TRAILING_HASH
• REMOVE_WWW
• REPLACE_IP_WITH_DOMAIN_NAME
• SECURE_SCHEME
• SORT_QUERY_PARAMETERS
• UNSECURE_SCHEME
• UPPERCASE_ESCAPESEQUENCE