What's new​

One config, three formats. XML, YAML, and JSON are equally supported — pick whichever your team already works in. The same crawler, in two of them:

id: acme-website
startReferences:
  - https://www.example.com
maxDepth: 5
committers:
  - class: ElasticsearchCommitter
    indexName: web-content

{
  "id": "acme-website",
  "startReferences": ["https://www.example.com"],
  "maxDepth": 5,
  "committers": [
    { "class": "ElasticsearchCommitter", "indexName": "web-content" }
  ]
}

One handler pipeline. v3 split content and metadata work across taggers, transformers, and filters. In v4 those merged: transformers handle both, and filters became conditions you can combine with allOf, anyOf, and noneOf inside the same handler list. Fewer concepts to hold in your head, not fewer capabilities.

Runs across more than one machine. A new cluster connector layer — Hazelcast, MVStore (the default), or in-memory — lets a single crawl be worked by several nodes against a shared queue. Point it at Hazelcast and scale a crawl out horizontally; leave it on the default and nothing changes.

Broader reach out of the box. The file system crawler's fetchers were rebuilt on Java's own NIO2 file system API instead of Apache Commons VFS2, and the list grew alongside it: S3, Azure Blob, Azure Data Lake Gen2, Google Cloud Storage, Google Drive, SharePoint and OneDrive via Microsoft 365 Graph, Box, Egnyte, CMIS repositories, HDFS, SFTP, FTP, SMB, WebDAV, and archive files. On the web side, Playwright joins WebDriver for browser-based crawling, driving real Chromium, Firefox, and WebKit engines without a separate driver binary to manage.

14 committers, one version number. Elasticsearch, Solr, OpenText IDOL, Amazon CloudSearch, Google Cloud Search, Microsoft Azure Search, SQL, Neo4j, Apache Kafka, plus CSV, JSON, and XML file output — all in the same repository as the crawler, versioned together. No more chasing a committer release that lags the core.

That is a partial list. The reference documentation below is the complete one.

The Visual Configurator​

Alongside the crawler, we built the Configurator — a browser-based tool for building, editing, and validating crawler configurations without hand-writing a config file. Import an existing config or start from scratch, edit visually, and export as XML, YAML, or JSON.

A few things worth knowing about it:

• It's free. No account required to use the core editing and export workflow.
• It is not open source. The crawler is, and stays that way. The Configurator is a separate, hosted tool Norconex builds and maintains alongside it.
• Nothing you paste leaves your browser. Conversion and validation run entirely client-side — which matters, because real configuration files routinely contain credentials, internal hostnames, and proxy settings.
• It understands v3. Paste in a v3 XML config and the Configurator detects it automatically, converts it — splitting a multi-crawler v3 file into one v4 configuration per crawler — and hands back a migration report listing what changed and what needs a second look.

If you're coming from v3, this is the fastest way to see your own configuration on the other side of the rewrite.

Reference docs you can edit​

Every configurable class in the crawler — every fetcher, committer, transformer, condition, and the rest — gets a reference page. Over 230 of them, covering the crawler core, the importer, and every committer.

Each page is two things merged at build time. The schema generates the structure: every property a class accepts, its type, and how it's validated, pulled straight from the crawler's own configuration model — that part cannot drift, since it either matches the code or the build fails. The descriptions, the notes on when you'd actually reach for an option, and the worked examples are written by hand, as MDX files living in docs/reference-source inside the crawler repository itself, not this website's.

That last part is new with v4: a meaningful share of what you're reading on this site is now part of the open-source project, versioned and reviewed the same way the code is. Every reference page carries a real Edit this page link that goes straight to the source file behind it. If a description is unclear, an example is missing, or something is just wrong, that's a pull request, not a support ticket — you're welcome to send one.

The migration guide and the Configurator both link into these pages directly, so wherever you land, you're one click from the exact options a class supports right now, and one more click from fixing it if it's wrong.

This is a beta — please try to break it​

Beta means what it says: we're not adding features, we're finding out where this one breaks. That's a different job than v3 has been doing for years in production, and no amount of internal testing substitutes for configurations we didn't write ourselves running against sites and file shares we don't control.

A few ways to help, roughly in order of how much we'd appreciate it:

1. Run a real v3 config through it. If you're already a v3 user, the highest-value thing you can do is convert one of your actual configurations with the Configurator and point it at a copy of your usual source. That's the fastest way for us to see configuration shapes we didn't anticipate.
2. Crawl something unusual. An oddly structured site, a file share with a directory depth nobody planned for, an authentication flow we haven't tested — this is exactly the kind of thing a beta is for.
3. File what you find. Bugs, confusing behavior, a documentation page that doesn't match reality — open an issue on GitHub and tag it with the V4 label so it doesn't get lost among v3 reports. For anything less clear-cut, start a discussion instead.

We'll be watching both.

Try it​

• Download the ZIP or pull the Docker image — Java 21 is the only prerequisite, and it's Apache 2.0 either way.
• Getting Started walks through installation, Docker, and a first crawl for both the web and file system crawlers.
• Migrating from v3? starts with the Configurator's automatic conversion, then covers everything it can't do for you.

Thanks for testing this with us.

The name​

Nexie comes out of Norconex — the end of the company name, given a spider to carry it.

Why a dock spider​

If you have spent a summer near a lake in Canada, you have already met her relatives. Dock spiders — properly Dolomedes, the fishing spiders — are the big ones that sit on the boards at the end of a dock and disappear the moment you lean in for a closer look. Norconex is Canadian, so the choice was somewhat made for us.

Two things make them worth borrowing from.

They walk on water. The surface film does for them what a web does for other spiders: they rest their legs on it and read the vibrations travelling across it, and that is how they work out what is out there and what is worth going after.

And then they go under. Not a toe in the water — all the way under, hunting aquatic insects, tadpoles and even small fish, carrying a film of trapped air against their body so they can stay down there and keep working.

Cover the surface, feel out what matters, then go deep after it and come back with something. We did not have to stretch that one very far.

It also happens to describe the two directions a crawl gets pulled in.

Going wide is a scaling problem. A lot of sites, a lot of shares, a great many documents, and a light touch on each one — you are covering ground, and what you mostly need is for that to keep working at volume.

Going deep is a completely different problem. One source at a time, with filtering, parsing, transformation and enrichment shaped around that particular content and around what one organisation actually needs to get out of it. No two enterprises want the same things out of the same documents.

Those pull against each other, and it is tempting to build for one of them. Version 4 is the release where you stop having to choose. The same configuration model describes either, and a wide crawl and a deep one can sit side by side in a single file.

Thank you, Frank​

Nexie is not the first spider to represent this project. Her predecessor never had an official name — the team just called him Frank, and it stuck, the way those things do. He has been on the front of this crawler for a long time.

This is a thank-you and not a send-off. Frank has not been retired: version 3 is still supported and still running in production in plenty of places, and Frank is still its face. He has handed over the new stack, not the job.

The two of them are worth a second look side by side. Frank wears a cap with a magnifying glass on it — the crawler as the thing that goes out and finds what you asked for. Nexie carries cubes of data with binary streaming out of them — the crawler as the thing that takes what it found and delivers it somewhere useful. Eight legs each, both in Norconex blue, a different job description.

Thanks, Frank. Enjoy the shallow end.

Where you will see her​

Nexie is on the site, in the docs, and on every release from v4 forward. The artwork itself — and what you are welcome to do with it — lives on the Brand page.