Package com.norconex.crawler.core.cmd.crawl

Class RunSummaryWriter

java.lang.Object
com.norconex.crawler.core.cmd.crawl.RunSummaryWriter
public final class RunSummaryWriter extends Object
Writes a machine-readable summary of a crawl when the runSummaryFile system property names a file to write it to: 
     -DrunSummaryFile=/var/run/my-crawl/summary.json

The property is read rather than a configuration setting because the caller that wants the summary is usually the one that launched the JVM — a supervising process, a scheduler, a CI job — and it needs to choose the path without editing the crawl configuration. This mirrors -DenableJMX=true, which exists for the same reason. Nothing is written when the property is absent.

The file is produced at the end of a crawl, while the ledger is still open, and looks like this: 

 {
   "crawlerId" : "my-crawler",
   "state" : "COMPLETED",
   "counts" : { "NEW" : 3, "UNMODIFIED" : 41 }
 }

Counts are keyed by ProcessingOutcome name, so a reader gets the crawler's own vocabulary rather than a translation of it. Obtaining them costs one pass over the entries processed in this run, which is why this is opt-in rather than always on.

  • Field Details

    • SYS_PROP_RUN_SUMMARY_FILE

      public static final String SYS_PROP_RUN_SUMMARY_FILE
      System property naming the file to write the run summary to.
      See Also:

  • Method Details

    • writeIfRequested

      public static void writeIfRequested(CrawlerSession session, CrawlerState finalState)
      Writes the summary if the system property asks for one.

      A failure here is logged and swallowed. A report that could not be written is not a reason to fail a crawl that otherwise succeeded, and the crawl's real outcome is already recorded in the ledger and the exit code.

      Parameters:
      session - the crawl session, still open
      finalState - the state the crawl ended in