SegmentCountUrlFilter

Filter by the number of reference segments.

crawler/webJavadoc: SegmentCountUrlFilter

Notes

Filters URLs based on the number of path segments (or duplicate segments) in the reference. URLs exceeding the configured segment count threshold are included or excluded depending on the onMatch setting.

Examples

The above example will reject URLs with more than 5 forward slashes after the domain.

YAML

class: SegmentCountUrlFilter
onMatch: EXCLUDE
count: 5

JSON

{
  "class": "SegmentCountUrlFilter",
  "onMatch": "EXCLUDE",
  "count": 5
}

XML

<class>SegmentCountUrlFilter</class>
<onMatch>EXCLUDE</onMatch>
<count>5</count>

Usage

Full configuration skeleton, with every option and its default

YAML

class: SegmentCountUrlFilter
count: 0
duplicate: false
onMatch: INCLUDE
separator: string

JSON

{
  "class": "SegmentCountUrlFilter",
  "count": 0,
  "duplicate": false,
  "onMatch": "INCLUDE",
  "separator": "string"
}

XML

<segmentCountUrlFilter>
  <class>SegmentCountUrlFilter</class>
  <count>0</count>
  <duplicate>false</duplicate>
  <onMatch>INCLUDE</onMatch>
  <separator>string</separator>
</segmentCountUrlFilter>

Properties

Property	Type	Required	Default
count	integer	No	10
duplicate	boolean	No	false
onMatch	enum	No	INCLUDE
separator	string	No	/

Property Details

count

Minimum number of segments.

duplicate

Use this option to count the number of duplicate segments instead.

onMatch

What to do with a reference once its segment count has been reached.

Allowed Values

• INCLUDE
• EXCLUDE

separator

Regular expression matching an alternate segment separator (default is: /)