Build only what you need

Pick a crawler, a distribution, and the committers you actually use. You get a Maven POM that assembles that distribution, and a Dockerfile that packages it — no fork, no patching, no manual dependency hunting.

Ready-to-run packages

VersionChecking GitHub for the latest release…Looking for version 3.x or earlier? Visit opensource.norconex.com.

Crawler

Distribution

Standard leaves out optional features (clustering, OCR, ...). The generated POM lists each one — delete its exclusions to add it back.

Committers

Built-in committers (Log, JSON, XML, CSV, ...) are always included.

Elasticsearch and OpenSearchElasticsearch 7+ / OpenSearch via RESTApache SolrSolr / Lucidworks FusionAzure AI SearchFormerly Azure Cognitive SearchGoogle Cloud SearchGoogle Workspace / Cloud Search indexingOpenText IDOLMicro Focus / OpenText IDOLAmazon CloudSearchAmazon CloudSearch domainsNeo4jGraph database nodes & relationshipsSQL / JDBCAny JDBC databaseApache KafkaPublish to Kafka topics

Options

Playwright supportSecond browser engine, left out of the official distributions for its size. WebDriver rendering is already included.

Save as pom.xml and run mvn clean package — your tailored distribution (crawler, selected committers, and all dependencies) is assembled under target/crawler/lib/.

Checking the latest release…

Embedding the crawler in a Java app
Running the crawler from your own code, rather than packaging a distribution to launch from a shell.
What each committer does
Every destination the crawler can write to, and the reference documentation for each one.
Installing what you built
Where the assembled distribution lands, and how to launch it.

Learn

Getting Started
Features
Concepts
Configuration Reference
Migration: v3 → v4
Blog

Tools

Configuration Editor
Integrations
Download

Community

Support Plans
Sponsors
GitHub
Discussions
Issues
Norconex Inc.
Privacy Policy

Brand

Brand & assets

Crawler

Distribution

Committers

Options

Related