Extending the Crawler

Norconex Crawler is designed for extension. Every major component — parsers, filters, transformers, committers, fetchers — is a plugin point you can replace or augment without forking or modifying the core.

The simplest approach: drop in a class

The minimal path to a custom extension is three steps:

1. Implement the relevant interface
2. Package your class in a JAR and drop it in the lib/ folder of the distribution
3. Reference it by fully-qualified class name in your configuration

No annotation, no registration file, no code generation required. The configuration loader resolves any fully-qualified class name directly at runtime.

documentFilters:
  - class: com.example.PricingPageFilter

That is all that is needed for the crawler to instantiate and use your class.

Common extension points

Crawl-stage document filter

Implement DocumentFilter to accept or reject a fetched document before it enters the Importer pipeline.

package com.example;

import com.norconex.crawler.core.doc.operations.filter.DocumentFilter;
import com.norconex.importer.doc.Doc;

public class PricingPageFilter implements DocumentFilter {

    @Override
    public boolean acceptDocument(Doc document) {
        return document.getReference().contains("pricing");
    }
}

documentFilters:
  - class: com.example.PricingPageFilter

Importer handler

Implement DocHandler to add custom behavior inside the Importer pipeline — enriching metadata, transforming content, or rejecting a document. Return false to stop processing and discard the document.

package com.example;

import com.norconex.importer.handler.DocHandler;
import com.norconex.importer.handler.DocHandlerContext;

public class UppercaseTitleHandler implements DocHandler {

    @Override
    public boolean handle(DocHandlerContext ctx) throws java.io.IOException {
        var title = ctx.metadata().getString("title");
        if (title != null) {
            ctx.metadata().set("title", title.toUpperCase());
        }
        return true;
    }
}

handlers:
  - class: com.example.UppercaseTitleHandler

Custom committer

Extend AbstractCommitter to send documents to any destination.

package com.example;

import com.norconex.committer.core.AbstractCommitter;
import com.norconex.committer.core.DeleteRequest;
import com.norconex.committer.core.UpsertRequest;

public class MyApiCommitter extends AbstractCommitter<MyApiCommitterConfig> {

    @Override
    protected void doUpsert(UpsertRequest request) {
        myApiClient.index(
            request.getReference(),
            request.getContent(),
            request.getMetadata());
    }

    @Override
    protected void doDelete(DeleteRequest request) {
        myApiClient.delete(request.getReference());
    }
}

committers:
  - class: com.example.MyApiCommitter

Short class names via SPI (optional)

By default, you must use the fully-qualified class name in configuration. If you want to reference your class by its simple name (without the package prefix), register it through the Service Provider Interface (SPI).

Create a class that extends BasePolymorphicTypeProvider and registers your types:

package com.example.spi;

import com.norconex.commons.lang.bean.spi.BasePolymorphicTypeProvider;
import com.example.PricingPageFilter;
import com.norconex.crawler.core.doc.operations.filter.DocumentFilter;

public class MyExtensionsPtProvider extends BasePolymorphicTypeProvider {

    @Override
    protected void register(Registry registry) {
        registry.add(DocumentFilter.class, PricingPageFilter.class);
    }
}

Then declare it as a service in your JAR:

META-INF/services/com.norconex.commons.lang.bean.spi.PolymorphicTypeProvider

with the content:

com.example.spi.MyExtensionsPtProvider

After this, the short name works in configuration:

documentFilters:
  - class: PricingPageFilter

Event listeners

All crawler events are instances of CrawlerEvent. There are no separate classes per event type — the event type is identified by a string name checked with event.is(String).

To create a reusable listener, implement EventListener<Event> and register it in your config. Its single method is accept(Event event).

package com.example;

import com.norconex.commons.lang.event.Event;
import com.norconex.commons.lang.event.EventListener;
import com.norconex.crawler.core.event.CrawlerEvent;

public class MyAuditListener implements EventListener<Event> {

    @Override
    public void accept(Event event) {
        if (!(event instanceof CrawlerEvent e)) {
            return;
        }
        if (e.is(CrawlerEvent.DOCUMENT_FETCHED)) {
            log.info("Fetched: {}", e.getCrawlEntry().getReference());
        }
        if (e.is(CrawlerEvent.REJECTED_FILTER)) {
            log.warn("Filtered out: {}", e.getCrawlEntry().getReference());
        }
    }
}

Register it in your configuration file:

eventListeners:
  - class: com.example.MyAuditListener

If you only care about lifecycle events (start, stop, clean, error), extend the CrawlerLifeCycleListener adapter and override only the methods you need:

public class MyCrawlLifecycle extends CrawlerLifeCycleListener {

    @Override
    protected void onCrawlerCrawlBegin(CrawlerEvent event) {
        log.info("Crawl started: {}", event.getCrawlSession().getCrawlerId());
    }

    @Override
    protected void onCrawlerCrawlEnd(CrawlerEvent event) {
        log.info("Crawl finished.");
    }
}

Selected event name constants on CrawlerEvent:

Constant	When it fires
CRAWLER_CRAWL_BEGIN	Crawl is about to begin
CRAWLER_CRAWL_END	Crawl completed normally
CRAWLER_ERROR	An error occurred in the crawler
DOCUMENT_FETCHED	A document was successfully fetched
DOCUMENT_IMPORTED	The Importer pipeline completed for a document
DOCUMENT_PROCESSED	A document finished processing (success or not)
REJECTED_FILTER	Discarded by a reference or document filter
REJECTED_NOTFOUND	Discarded because it no longer exists
REJECTED_ERROR	Discarded due to a processing error

For attaching listeners programmatically when embedding the crawler in Java, see Java Integration.

Packaging

Place your JAR in the lib/ directory of the crawler distribution. No changes to the core are required. For Maven projects, declare a dependency on the crawler module and the importer module — your JAR will be picked up automatically at runtime.

Class scanning

Everything above works by name: a fully-qualified class name is loaded directly, and an SPI provider registers its types explicitly. Neither needs the classpath to be searched, so neither is affected by anything in this section.

Scanning only comes in when something has to discover types it was not told about — addFromScan in a PolymorphicTypeProvider, or resolving a partial class name in a legacy XML configuration. That means opening every JAR on the classpath and reading its entries, and on a full distribution that is over 240 archives, nearly all of which hold nothing relevant.

Three settings control it. All are optional, and with none of them set behaviour is exactly as it has always been.

Restricting which JARs are scanned

norconex.classfinder.includeJars takes comma-separated wildcard patterns matched against JAR file names. A JAR that does not match is never opened:

-Dnorconex.classfinder.includeJars=nx-*.jar,norconex-*.jar

This is the setting that saves real time, because it decides from the file name without touching the archive. On the web crawler distribution it cuts a steady-state scan of the 242-entry classpath from roughly 190 ms to 60 ms, resolving the same types.

Directories are never excluded by these patterns, so working from target/classes or an IDE keeps behaving normally.

The extension directory

The ext/ directory — or wherever norconex.classfinder.extDir points — is always scanned when it exists, whatever the patterns say. It is where a JAR belongs when its types must be discovered by scanning but its name is excluded by the patterns above.

The launcher must also put it on the classpath for those classes to be loadable:

-cp "./lib/*:./ext/*"

Shipping a class index

A JAR may carry a META-INF/norconex/classes.idx resource listing the classes it holds, one per line, written either as class names (com.example.Foo) or as archive paths (com/example/Foo.class). Blank lines and lines starting with # are ignored. When present it is read instead of the archive being walked.

An index avoids enumerating a large JAR, but it does not avoid opening it — establishing whether an index exists costs the same file access either way. Prefer includeJars for speed and treat the index as a way for a JAR to declare its contents precisely.

Setting norconex.classfinder.scanMode=indexed additionally skips any classpath JAR that carries no index. Directories are still scanned.

Resources

• Reference — all built-in extension points with examples
• GitHub: Norconex/crawler — source code and existing implementations to use as a reference