Package com.norconex.crawler.web.doc.operations.link.impl

Class HtmlLinkExtractorConfig

java.lang.Object
com.norconex.crawler.web.doc.operations.link.impl.HtmlLinkExtractorConfig
public class HtmlLinkExtractorConfig extends Object

Configuration for HtmlLinkExtractor.

  • Field Details

    • DEFAULT_MAX_URL_LENGTH

      public static final int DEFAULT_MAX_URL_LENGTH
      Default maximum length a URL can have.
      See Also:

    • DEFAULT_SCHEMES

      public static final List<String> DEFAULT_SCHEMES
      Default supported URL schemes (http, https, and ftp).

    • HTTP_EQUIV

      public static final String HTTP_EQUIV
      See Also:

  • Constructor Details

    • HtmlLinkExtractorConfig

      public HtmlLinkExtractorConfig()

  • Method Details

    • getExtractBetweens

      public List<HtmlLinkExtractorConfig.RegexPair> getExtractBetweens()
      Gets the patterns delimiting the portions of a document to be considered for link extraction.
      Returns:
      extract between patterns

    • setExtractBetweens

      public HtmlLinkExtractorConfig setExtractBetweens(List<HtmlLinkExtractorConfig.RegexPair> betweens)
      Sets the patterns delimiting the portions of a document to be considered for link extraction.
      Parameters:
      betweens - extract between patterns
      Returns:
      this

    • addExtractBetween

      public HtmlLinkExtractorConfig addExtractBetween(String start, String end, boolean ignoreCase)
      Adds patterns delimiting a portion of a document to be considered for link extraction.
      Parameters:
      start - pattern matching start of text portion
      end - pattern matching end of text portion
      ignoreCase - whether the patterns are case sensitive or not
      Returns:
      this

    • getNoExtractBetweens

      public List<HtmlLinkExtractorConfig.RegexPair> getNoExtractBetweens()
      Gets the patterns delimiting the portions of a document to be excluded from link extraction.
      Returns:
      extract between patterns

    • setNoExtractBetweens

      public HtmlLinkExtractorConfig setNoExtractBetweens(List<HtmlLinkExtractorConfig.RegexPair> betweens)
      Sets the patterns delimiting the portions of a document to be excluded from link extraction.
      Parameters:
      betweens - extract between patterns
      Returns:
      this

    • addNoExtractBetween

      public HtmlLinkExtractorConfig addNoExtractBetween(String start, String end, boolean ignoreCase)
      Adds patterns delimiting a portion of a document to be excluded from link extraction.
      Parameters:
      start - pattern matching start of text portion
      end - pattern matching end of text portion
      ignoreCase - whether the patterns are case sensitive or not
      Returns:
      this

    • getExtractSelectors

      public List<String> getExtractSelectors()
      Gets the selectors matching the portions of a document to be considered for link extraction.
      Returns:
      selectors

    • setExtractSelectors

      public HtmlLinkExtractorConfig setExtractSelectors(List<String> selectors)
      Sets the selectors matching the portions of a document to be considered for link extraction.
      Parameters:
      selectors - selectors
      Returns:
      this

    • addExtractSelectors

      public HtmlLinkExtractorConfig addExtractSelectors(List<String> selectors)
      Adds selectors matching the portions of a document to be considered for link extraction.
      Parameters:
      selectors - selectors
      Returns:
      this

    • getNoExtractSelectors

      public List<String> getNoExtractSelectors()
      Gets the selectors matching the portions of a document to be excluded from link extraction.
      Returns:
      selectors

    • setNoExtractSelectors

      public HtmlLinkExtractorConfig setNoExtractSelectors(List<String> selectors)
      Sets the selectors matching the portions of a document to be excluded from link extraction.
      Parameters:
      selectors - selectors
      Returns:
      this

    • addNoExtractSelectors

      public HtmlLinkExtractorConfig addNoExtractSelectors(List<String> selectors)
      Adds selectors matching the portions of a document to be excluded from link extraction.
      Parameters:
      selectors - selectors
      Returns:
      this

    • getSchemes

      public List<String> getSchemes()
      Gets the schemes to be extracted.
      Returns:
      schemes to be extracted

    • setSchemes

      public HtmlLinkExtractorConfig setSchemes(List<String> schemes)
      Sets the schemes to be extracted.
      Parameters:
      schemes - schemes to be extracted
      Returns:
      this

    • setFieldMatcher

      public HtmlLinkExtractorConfig setFieldMatcher(TextMatcher fieldMatcher)

    • setContentTypeMatcher

      public HtmlLinkExtractorConfig setContentTypeMatcher(TextMatcher matcher)
      The matcher of content types to apply link extraction on. No attempt to extract links from any other content types will be made. Default is CommonMatchers.HTML_CONTENT_TYPES.
      Parameters:
      matcher - content type matcher
      Returns:
      this

    • clearRestrictions

      public void clearRestrictions()
      Clears all restrictions.

    • getRestrictions

      public PropertyMatchers getRestrictions()
      Gets all restrictions
      Returns:
      the restrictions

    • addLinkTag

      public HtmlLinkExtractorConfig addLinkTag(String tagName, String attribute)

    • removeLinkTag

      public HtmlLinkExtractorConfig removeLinkTag(String tagName, String attribute)

    • clearLinkTags

      public void clearLinkTags()

    • getContentTypeMatcher

      public TextMatcher getContentTypeMatcher()
      The matcher of content types to apply link extraction on. No attempt to extract links from any other content types will be made. Default is CommonMatchers.HTML_CONTENT_TYPES.

    • getFieldMatcher

      public TextMatcher getFieldMatcher()
      Matcher of one or more fields to use as the source of content to extract links from, instead of the document content.

    • getMaxURLLength

      public int getMaxURLLength()
      The maximum supported URL length. Longer URLs are ignored.

    • isIgnoreNofollow

      public boolean isIgnoreNofollow()
      Whether to ignore "nofollow" directives on HTML links. An example of such links: 
       <a href="https://yoursite.com/doNotCrawl.html" rel="nofollow">
   By default this link won't be crawled.
 </a>

    • isIgnoreLinkData

      public boolean isIgnoreLinkData()
      Gets whether to ignore extra data associated with a link.

    • getCharset

      public Charset getCharset()
      The character set to use for pages on which link extraction is performed. When null (default), character set detection will be attempted.

    • isCommentsEnabled

      public boolean isCommentsEnabled()
      Gets whether links should be extracted from comments. Comment example: 
       <!--
 By default, this URL won't be crawled:
 <a href="https://yoursite.com/somepage.html">Some URL</a>
 -->

    • getTagAttribs

      public Properties getTagAttribs()

    • setMaxURLLength

      public HtmlLinkExtractorConfig setMaxURLLength(int maxURLLength)
      The maximum supported URL length. Longer URLs are ignored.
      Returns:
      this.

    • setIgnoreNofollow

      public HtmlLinkExtractorConfig setIgnoreNofollow(boolean ignoreNofollow)
      Whether to ignore "nofollow" directives on HTML links. An example of such links: 
       <a href="https://yoursite.com/doNotCrawl.html" rel="nofollow">
   By default this link won't be crawled.
 </a>
      Returns:
      this.

    • setIgnoreLinkData

      public HtmlLinkExtractorConfig setIgnoreLinkData(boolean ignoreLinkData)
      Gets whether to ignore extra data associated with a link.
      Returns:
      this.

    • setCharset

      public HtmlLinkExtractorConfig setCharset(Charset charset)
      The character set to use for pages on which link extraction is performed. When null (default), character set detection will be attempted.
      Returns:
      this.

    • setCommentsEnabled

      public HtmlLinkExtractorConfig setCommentsEnabled(boolean commentsEnabled)
      Gets whether links should be extracted from comments. Comment example: 
       <!--
 By default, this URL won't be crawled:
 <a href="https://yoursite.com/somepage.html">Some URL</a>
 -->
      Returns:
      this.

    • equals

      public boolean equals(Object o)
      Overrides:
      equals in class Object

    • canEqual

      protected boolean canEqual(Object other)

    • hashCode

      public int hashCode()
      Overrides:
      hashCode in class Object

    • toString

      public String toString()
      Overrides:
      toString in class Object