GenericCanonicalLinkDetector

Replace all URLs encountered with their canonical version (when applicable).

crawler/webJavadoc: GenericCanonicalLinkDetector

Notes

Detects canonical URLs from HTTP response headers (Link: rel="canonical") and HTML <link rel="canonical"> tags. Non-canonical documents are replaced by their canonical equivalents. Optionally restricted to specific content types.

Usage

YAML

class: GenericCanonicalLinkDetector
contentTypes:
  - string

JSON

{
  "class": "GenericCanonicalLinkDetector",
  "contentTypes": [
    "string"
  ]
}

XML

<canonicalLinkDetector>
  <class>GenericCanonicalLinkDetector</class>
  <contentTypes>
    <contentType>string</contentType>
  </contentTypes>
</canonicalLinkDetector>

Properties

Property	Type	Required	Default
contentTypes	string[]	No	-

Property Details

contentTypes

One or more content types (e.g., text/html) for which to enable canonical link detection. Leave empty to enable detection for all content types.