Package com.norconex.committer.core.batch.queue.impl

Class FsQueue

java.lang.Object
com.norconex.committer.core.batch.queue.impl.FsQueue
All Implemented Interfaces:
CommitterQueue, Configurable<FsQueueConfig>, AutoCloseable
public class FsQueue extends Object implements CommitterQueue, Configurable<FsQueueConfig>

File System queue. Queues committer requests as zip files and deletes successfully committed zip files.

The top-level queue directory is the one defined in the CommitterContext initialization argument or the system temporary directory if null. A "queue" sub-folder will be created for queued requests, while an "error" one will also be created for failed batches.

Handling Failures

Before moving a failed batch to the "error" folder you have the opportunity to have this queue tell the Committer to retry the batch. This can be particularly useful if you experience issues such as:

  • The target repository momentarily failed to respond.
  • A specific document was rejected by the target repository.
  • The batch was too big for the repository to handle.
  • Etc.

In addition to specifying how many time to retry and how long to wait between each attempts, you can also break the batch size into smaller chunks. The "splitBatch" setting offers the following options:

  • OFF - Default. The failing batch is not split.
  • HALF - The batch is split in half at each failure (after maxRetries is reached), up until batches are of size 1 (committing documents individually).
  • ONE - Documents from the failing batch are sent one by one.

When "splitBatch" is not OFF and used in combination with a non-zero "maxRetry", the later then represents how many times to retry each new smaller batches created.