Package com.norconex.crawler.web.ledger

Class WebCrawlerEntry

java.lang.Object
com.norconex.crawler.core.ledger.CrawlerEntry
com.norconex.crawler.web.ledger.WebCrawlerEntry
All Implemented Interfaces:
Serializable
public class WebCrawlerEntry extends CrawlerEntry
A URL being crawled holding relevant crawl information.
See Also:

  • Constructor Details

    • WebCrawlerEntry

      public WebCrawlerEntry()

    • WebCrawlerEntry

      public WebCrawlerEntry(WebCrawlerEntry src)
      Copy constructor.
      Parameters:
      src - the source to copy from

    • WebCrawlerEntry

      public WebCrawlerEntry(String reference)

    • WebCrawlerEntry

      public WebCrawlerEntry(String url, int depth)
      Constructor.
      Parameters:
      url - URL being crawled
      depth - URL depth

  • Method Details

    • setReference

      public final void setReference(String url)
      Overrides:
      setReference in class CrawlerEntry

    • getUrlRoot

      public String getUrlRoot()
      Gets the URL root (protocol + domain, e.g. http://www.host.com).
      Returns:
      URL root

    • getReferencedUrls

      public List<String> getReferencedUrls()
      Gets URLs referenced by this one.
      Returns:
      URLs referenced by this one (never null).
      Since:
      2.6.0

    • setReferencedUrls

      public void setReferencedUrls(List<String> referencedUrls)
      Sets URLs referenced by this one.
      Parameters:
      referencedUrls - referenced URLs
      Since:
      3.0.0

    • getRedirectTrail

      public List<String> getRedirectTrail()
      Gets the trail of URLs that were redirected up to this one.
      Returns:
      URL redirection trail to this one (never null).
      Since:
      2.8.0

    • setRedirectTrail

      public void setRedirectTrail(List<String> redirectTrail)
      Sets the trail of URLs that were redirected up to this one.
      Parameters:
      redirectTrail - URL redirection trail to this one
      Since:
      3.0.0

    • addRedirectURL

      public void addRedirectURL(String url)
      Adds a redirect URL to the trail of URLs that were redirected so far.
      Parameters:
      url - URL to add
      Since:
      3.0.0

    • isFromSitemap

      public boolean isFromSitemap()
      Whether this record was obtained from parsing a sitemap. Set by the queue pipeline so implementors should not have to set it themselves.

    • getSitemapLastMod

      public ZonedDateTime getSitemapLastMod()
      The document last modified date according to sitemap.

    • getSitemapChangeFreq

      public String getSitemapChangeFreq()
      The document change frequency according to sitemap.

    • getSitemapPriority

      public Float getSitemapPriority()
      The document priority according to sitemap.

    • getReferrerReference

      public String getReferrerReference()

    • getReferrerLinkMetadata

      public String getReferrerLinkMetadata()

    • getEtag

      public String getEtag()
      The HTTP ETag.
      Since:
      3.0.0

    • getRedirectTarget

      public String getRedirectTarget()

    • getHttpStatusCode

      public int getHttpStatusCode()
      HTTP status code from the last fetch response.

    • getHttpReasonPhrase

      public String getHttpReasonPhrase()
      HTTP reason phrase from the last fetch response.

    • setUrlRoot

      public void setUrlRoot(String urlRoot)

    • setFromSitemap

      public void setFromSitemap(boolean fromSitemap)
      Whether this record was obtained from parsing a sitemap. Set by the queue pipeline so implementors should not have to set it themselves.

    • setSitemapLastMod

      public void setSitemapLastMod(ZonedDateTime sitemapLastMod)
      The document last modified date according to sitemap.

    • setSitemapChangeFreq

      public void setSitemapChangeFreq(String sitemapChangeFreq)
      The document change frequency according to sitemap.

    • setSitemapPriority

      public void setSitemapPriority(Float sitemapPriority)
      The document priority according to sitemap.

    • setReferrerReference

      public void setReferrerReference(String referrerReference)

    • setReferrerLinkMetadata

      public void setReferrerLinkMetadata(String referrerLinkMetadata)

    • setEtag

      public void setEtag(String etag)
      The HTTP ETag.
      Since:
      3.0.0

    • setRedirectTarget

      public void setRedirectTarget(String redirectTarget)

    • setHttpStatusCode

      public void setHttpStatusCode(int httpStatusCode)
      HTTP status code from the last fetch response.

    • setHttpReasonPhrase

      public void setHttpReasonPhrase(String httpReasonPhrase)
      HTTP reason phrase from the last fetch response.

    • equals

      public boolean equals(Object o)
      Overrides:
      equals in class CrawlerEntry

    • canEqual

      protected boolean canEqual(Object other)
      Overrides:
      canEqual in class CrawlerEntry

    • hashCode

      public int hashCode()
      Overrides:
      hashCode in class CrawlerEntry

    • toString

      public String toString()
      Overrides:
      toString in class CrawlerEntry