Skip to main content

Norconex Crawler v4 Beta Is Here

· 6 min read
Norconex Team
Norconex Team
Norconex Inc.

Norconex Crawler v4 is feature-complete and ready for a beta release. It is a rewrite, not a patch on v3 — new configuration model, new connector lineup, a free visual configurator, and documentation generated straight from the schema instead of hand-maintained alongside it. We think it holds up. Now we want a lot of configurations we did not write pointed at it, so tell us where it doesn't.

What's new​

One config, three formats. XML, YAML, and JSON are equally supported — pick whichever your team already works in. The same crawler, in two of them:

id: acme-website
startReferences:
  - https://www.example.com
maxDepth: 5
committers:
  - class: ElasticsearchCommitter
    indexName: web-content

One handler pipeline. v3 split content and metadata work across taggers, transformers, and filters. In v4 those merged: transformers handle both, and filters became conditions you can combine with allOf, anyOf, and noneOf inside the same handler list. Fewer concepts to hold in your head, not fewer capabilities.

Runs across more than one machine. A new cluster connector layer — Hazelcast, MVStore (the default), or in-memory — lets a single crawl be worked by several nodes against a shared queue. Point it at Hazelcast and scale a crawl out horizontally; leave it on the default and nothing changes.

Broader reach out of the box. The file system crawler's fetchers were rebuilt on Java's own NIO2 file system API instead of Apache Commons VFS2, and the list grew alongside it: S3, Azure Blob, Azure Data Lake Gen2, Google Cloud Storage, Google Drive, SharePoint and OneDrive via Microsoft 365 Graph, Box, Egnyte, CMIS repositories, HDFS, SFTP, FTP, SMB, WebDAV, and archive files. On the web side, Playwright joins WebDriver for browser-based crawling, driving real Chromium, Firefox, and WebKit engines without a separate driver binary to manage.

14 committers, one version number. Elasticsearch, Solr, OpenText IDOL, Amazon CloudSearch, Google Cloud Search, Microsoft Azure Search, SQL, Neo4j, Apache Kafka, plus CSV, JSON, and XML file output — all in the same repository as the crawler, versioned together. No more chasing a committer release that lags the core.

That is a partial list. The reference documentation below is the complete one.

The Visual Configurator​

Alongside the crawler, we built the Configurator — a browser-based tool for building, editing, and validating crawler configurations without hand-writing a config file. Import an existing config or start from scratch, edit visually, and export as XML, YAML, or JSON.

A few things worth knowing about it:

  • It's free. No account required to use the core editing and export workflow.
  • It is not open source. The crawler is, and stays that way. The Configurator is a separate, hosted tool Norconex builds and maintains alongside it.
  • Nothing you paste leaves your browser. Conversion and validation run entirely client-side — which matters, because real configuration files routinely contain credentials, internal hostnames, and proxy settings.
  • It understands v3. Paste in a v3 XML config and the Configurator detects it automatically, converts it — splitting a multi-crawler v3 file into one v4 configuration per crawler — and hands back a migration report listing what changed and what needs a second look.

If you're coming from v3, this is the fastest way to see your own configuration on the other side of the rewrite.

Reference docs you can edit​

Every configurable class in the crawler — every fetcher, committer, transformer, condition, and the rest — gets a reference page. Over 230 of them, covering the crawler core, the importer, and every committer.

Each page is two things merged at build time. The schema generates the structure: every property a class accepts, its type, and how it's validated, pulled straight from the crawler's own configuration model — that part cannot drift, since it either matches the code or the build fails. The descriptions, the notes on when you'd actually reach for an option, and the worked examples are written by hand, as MDX files living in docs/reference-source inside the crawler repository itself, not this website's.

That last part is new with v4: a meaningful share of what you're reading on this site is now part of the open-source project, versioned and reviewed the same way the code is. Every reference page carries a real Edit this page link that goes straight to the source file behind it. If a description is unclear, an example is missing, or something is just wrong, that's a pull request, not a support ticket — you're welcome to send one.

The migration guide and the Configurator both link into these pages directly, so wherever you land, you're one click from the exact options a class supports right now, and one more click from fixing it if it's wrong.

This is a beta — please try to break it​

Beta means what it says: we're not adding features, we're finding out where this one breaks. That's a different job than v3 has been doing for years in production, and no amount of internal testing substitutes for configurations we didn't write ourselves running against sites and file shares we don't control.

A few ways to help, roughly in order of how much we'd appreciate it:

  1. Run a real v3 config through it. If you're already a v3 user, the highest-value thing you can do is convert one of your actual configurations with the Configurator and point it at a copy of your usual source. That's the fastest way for us to see configuration shapes we didn't anticipate.
  2. Crawl something unusual. An oddly structured site, a file share with a directory depth nobody planned for, an authentication flow we haven't tested — this is exactly the kind of thing a beta is for.
  3. File what you find. Bugs, confusing behavior, a documentation page that doesn't match reality — open an issue on GitHub and tag it with the V4 label so it doesn't get lost among v3 reports. For anything less clear-cut, start a discussion instead.

We'll be watching both.

Try it​

  • Download the ZIP or pull the Docker image — Java 21 is the only prerequisite, and it's Apache 2.0 either way.
  • Getting Started walks through installation, Docker, and a first crawl for both the web and file system crawlers.
  • Migrating from v3? starts with the Configurator's automatic conversion, then covers everything it can't do for you.

Thanks for testing this with us.