ConditionMetadataFilter
Filter by a condition on metadata field values: text, number (e.g., a size), date (e.g., a modification date) or blank, optionally combined.
Notes
Accepts or rejects a document based on a condition evaluated on its metadata. Unlike GenericMetadataFilter, which only matches text, this filter can compare numbers and dates, and can combine several tests with allOf, anyOf and noneOf groups, nested as deep as needed.
Only metadata is ever tested, never the document content. A text, number, date or blank condition must therefore name the field(s) it applies to.
A condition that cannot be evaluated is simply not met: a missing field, a value that is not a number, or a value that is not a date in the expected format never matches. Combined with an onMatch of EXCLUDE, such documents are kept. Use a blank condition if you need to act on a missing field.
Filtering before download
This filter is most useful as a metadata filter that runs before a document is downloaded, to avoid downloading what you do not want (for instance, very large files). That only happens when the crawler obtains metadata in a separate step first: see metadataFetchSupport in the crawler configuration.
- Web crawler: metadata comes from an HTTP
HEADrequest, which means two requests per page instead of one. Servers do not always returnContent-LengthorLast-Modified(dynamic pages often do not), in which case the condition is not met. LeavemetadataFetchSupportdisabled unless you have good reason to do otherwise, such as skipping huge files you would rather not download. - File crawlers: metadata comes from the file attributes (size, last modified, and so on), which can be inexpensive for local files, but may cost a round-trip per file for remote storage. The
metadataFiltersof a file crawler only judge files, never folders, so a size condition does not reject the folders leading to the files. To decide which folders to open, for instance by their dates, use this filter infolderMetadataFilters(a folder has no size, but it hascrawler.lastModified,crawler.creationTimeandcrawler.hidden).
When metadataFetchSupport is disabled, this filter, as any metadata filter, still runs but after the document was downloaded, on the metadata obtained with it.
Examples
Skip files bigger than 10 MB
The Content-Length field holds a size in bytes.
- YAML
- JSON
- XML
class: ConditionMetadataFilter
onMatch: EXCLUDE
condition:
class: MetadataNumberCondition
fieldMatcher:
pattern: Content-Length
valueMatcher:
operator: gt
number: 10000000
{
"class": "ConditionMetadataFilter",
"onMatch": "EXCLUDE",
"condition": {
"class": "MetadataNumberCondition",
"fieldMatcher": {
"pattern": "Content-Length"
},
"valueMatcher": {
"operator": "gt",
"number": 10000000
}
}
}
<class>ConditionMetadataFilter</class>
<onMatch>EXCLUDE</onMatch>
<condition>
<class>MetadataNumberCondition</class>
<fieldMatcher>
<pattern>Content-Length</pattern>
</fieldMatcher>
<valueMatcher>
<operator>gt</operator>
<number>10000000</number>
</valueMatcher>
</condition>
Skip documents not modified for two years, or that are videos
- YAML
- JSON
- XML
class: ConditionMetadataFilter
onMatch: EXCLUDE
condition:
class: MetadataAnyOf
conditions:
- class: MetadataDateCondition
fieldMatcher:
pattern: Last-Modified
format: EEE, dd MMM yyyy HH:mm:ss zzz
valueMatcher:
operator: lt
date: TODAY-2Y
- class: MetadataTextCondition
fieldMatcher:
pattern: Content-Type
valueMatcher:
method: WILDCARD
pattern: video/*
{
"class": "ConditionMetadataFilter",
"onMatch": "EXCLUDE",
"condition": {
"class": "MetadataAnyOf",
"conditions": [
{
"class": "MetadataDateCondition",
"fieldMatcher": {
"pattern": "Last-Modified"
},
"format": "EEE, dd MMM yyyy HH:mm:ss zzz",
"valueMatcher": {
"operator": "lt",
"date": "TODAY-2Y"
}
},
{
"class": "MetadataTextCondition",
"fieldMatcher": {
"pattern": "Content-Type"
},
"valueMatcher": {
"method": "WILDCARD",
"pattern": "video/*"
}
}
]
}
}
<class>ConditionMetadataFilter</class>
<onMatch>EXCLUDE</onMatch>
<condition>
<class>MetadataAnyOf</class>
<conditions>
<class>MetadataDateCondition</class>
<fieldMatcher>
<pattern>Last-Modified</pattern>
</fieldMatcher>
<format>EEE, dd MMM yyyy HH:mm:ss zzz</format>
<valueMatcher>
<operator>lt</operator>
<date>TODAY-2Y</date>
</valueMatcher>
<class>MetadataTextCondition</class>
<fieldMatcher>
<pattern>Content-Type</pattern>
</fieldMatcher>
<valueMatcher>
<method>WILDCARD</method>
<pattern>video/*</pattern>
</valueMatcher>
</conditions>
</condition>
Usage
Full configuration skeleton, with every option and its default
- YAML
- JSON
- XML
class: ConditionMetadataFilter
condition:
class: MetadataNumberCondition
fieldMatcher:
ignoreCase: false
ignoreDiacritic: false
matchEmpty: false
method: BASIC
negateMatches: false
partial: false
pattern: string
replaceAll: false
trim: false
valueMatcher:
number: 0
operator: gt
valueMatcherRangeEnd:
number: 0
operator: gt
onMatch: INCLUDE
{
"class": "ConditionMetadataFilter",
"condition": {
"class": "MetadataNumberCondition",
"fieldMatcher": {
"ignoreCase": false,
"ignoreDiacritic": false,
"matchEmpty": false,
"method": "BASIC",
"negateMatches": false,
"partial": false,
"pattern": "string",
"replaceAll": false,
"trim": false
},
"valueMatcher": {
"number": 0,
"operator": "gt"
},
"valueMatcherRangeEnd": {
"number": 0,
"operator": "gt"
}
},
"onMatch": "INCLUDE"
}
<conditionMetadataFilter>
<class>ConditionMetadataFilter</class>
<condition>
<class>MetadataNumberCondition</class>
<fieldMatcher/>
<valueMatcher/>
<valueMatcherRangeEnd/>
</condition>
<onMatch>INCLUDE</onMatch>
</conditionMetadataFilter>
Properties
| Property | Type | Required | Default |
|---|---|---|---|
condition | MetadataCondition | No | - |
onMatch | enum | No | INCLUDE |
Property Details
condition
The condition to test. Either a single test (MetadataTextCondition, MetadataNumberCondition, MetadataDateCondition or MetadataBlankCondition) or a group of conditions (MetadataAllOf, MetadataAnyOf or MetadataNoneOf). When not set, every document is accepted.
onMatch
What to do with a document for which the condition is met. EXCLUDE rejects it; INCLUDE (the default) keeps it and rejects the documents for which the condition is not met.
Remember that when at least one include filter exists, a document must satisfy at least one of them to be accepted.
Allowed Values
INCLUDEEXCLUDE