Package com.norconex.crawler.core.cmd.crawl

Class RunProgressWriter

java.lang.Object
com.norconex.crawler.core.cmd.crawl.RunProgressWriter
All Implemented Interfaces:
EventListener<Event>, EventListener, Consumer<Event>
public class RunProgressWriter extends Object implements EventListener<Event>
Writes crawl progress as newline-delimited JSON when the runProgressFile system property names a file to write it to: 
     -DrunProgressFile=/var/run/my-crawl/progress.ndjson

Like -DenableJMX=true and RunSummaryWriter.SYS_PROP_RUN_SUMMARY_FILE, this is a system property rather than a configuration setting, because the caller that wants the file is the one that launched the JVM — a supervising process, a scheduler, a CI job — and it should not have to edit the crawl configuration to ask for it. Nothing is written when the property is absent, and nothing about the crawl configuration changes when it is present.

What it writes

One JSON object per line, of two kinds: 

 {"type":"progress","crawlerId":"x","at":"2026-09-12T04:10:00Z","elapsedMs":12345,
  "queued":42,"processing":2,"processed":1203,"baseline":5000,
  "sessionEventCounts":{"COMMITTER_UPSERT_END":1200, ...},
  "runEventCounts":{"COMMITTER_UPSERT_END":1200, ...}}
 {"type":"heartbeat", ... same fields ... }

The four counters are gauges: current state, answering how far along the crawl is. They cannot be derived from any tally, since a cumulative count of queueings says nothing about how deep the queue is now. The two maps are counters, by the crawler's own event names, so nothing here is a selection made in advance — a new event type appears as a new key. They differ only for a crawl that was stopped and resumed: the session totals span its attempts, as the gauges do, while the run counts describe the attempt that wrote the line.

A progress line is written only when the numbers actually changed since the last one, so a crawl that is politely waiting between requests stays quiet. A heartbeat line is written when nothing has changed for a while, which is what separates "slow" from "stuck" for a supervisor that can otherwise only see that the process is alive.

Why sampling rather than listening

The counts come from CrawlerMetrics on a timer, not from an event fired per document. A crawl of a local file system with several threads can finalize thousands of documents a second, and a writer on that path would put file I/O in the crawl loop and produce a volume of output nobody can consume. Sampling costs nothing per document and bounds the output by wall-clock instead of by crawl speed — the faster the crawl, the more it compresses.

Note that in a clustered crawl each node writes its own file, since each has its own view to report.

  • Field Details

    • SYS_PROP_RUN_PROGRESS_FILE

      public static final String SYS_PROP_RUN_PROGRESS_FILE
      System property naming the file to write crawl progress to.
      See Also:

    • SYS_PROP_RUN_PROGRESS_INTERVAL

      public static final String SYS_PROP_RUN_PROGRESS_INTERVAL
      How often the counters are sampled. Optional, as a duration string.
      See Also:

    • SYS_PROP_RUN_HEARTBEAT_INTERVAL

      public static final String SYS_PROP_RUN_HEARTBEAT_INTERVAL
      How long without change before a heartbeat is written. Optional.
      See Also:

  • Method Details

    • registerIfRequested

      public static void registerIfRequested(EventManager eventManager)
      Registers a progress writer on the given event manager if the system property asks for one.

      Registering here rather than through the crawl configuration is the point: the crawler's configurable surface is unchanged, so a caller that compiles configurations does not have to know this exists.

      Parameters:
      eventManager - the event manager to register with

    • accept

      public void accept(Event event)
      Specified by:
      accept in interface Consumer<Event>