Meet Nexie
Version 4 changed enough of this crawler that the mascot on the front of it no longer described the thing behind it. So we drew a new one. Meet Nexie.
The name
Nexie comes out of Norconex — the end of the company name, given a spider to carry it.
Why a dock spider
If you have spent a summer near a lake in Canada, you have already met her relatives. Dock spiders — properly Dolomedes, the fishing spiders — are the big ones that sit on the boards at the end of a dock and disappear the moment you lean in for a closer look. Norconex is Canadian, so the choice was somewhat made for us.
Two things make them worth borrowing from.
They walk on water. The surface film does for them what a web does for other spiders: they rest their legs on it and read the vibrations travelling across it, and that is how they work out what is out there and what is worth going after.
And then they go under. Not a toe in the water — all the way under, hunting aquatic insects, tadpoles and even small fish, carrying a film of trapped air against their body so they can stay down there and keep working.
Cover the surface, feel out what matters, then go deep after it and come back with something. We did not have to stretch that one very far.
It also happens to describe the two directions a crawl gets pulled in.
Going wide is a scaling problem. A lot of sites, a lot of shares, a great many documents, and a light touch on each one — you are covering ground, and what you mostly need is for that to keep working at volume.
Going deep is a completely different problem. One source at a time, with filtering, parsing, transformation and enrichment shaped around that particular content and around what one organisation actually needs to get out of it. No two enterprises want the same things out of the same documents.
Those pull against each other, and it is tempting to build for one of them. Version 4 is the release where you stop having to choose. The same configuration model describes either, and a wide crawl and a deep one can sit side by side in a single file.
Thank you, Frank
Nexie is not the first spider to represent this project. Her predecessor never had an official name — the team just called him Frank, and it stuck, the way those things do. He has been on the front of this crawler for a long time.
This is a thank-you and not a send-off. Frank has not been retired: version 3 is still supported and still running in production in plenty of places, and Frank is still its face. He has handed over the new stack, not the job.
The two of them are worth a second look side by side. Frank wears a cap with a magnifying glass on it — the crawler as the thing that goes out and finds what you asked for. Nexie carries cubes of data with binary streaming out of them — the crawler as the thing that takes what it found and delivers it somewhere useful. Eight legs each, both in Norconex blue, a different job description.
Thanks, Frank. Enjoy the shallow end.
Where you will see her
Nexie is on the site, in the docs, and on every release from v4 forward. The artwork itself — and what you are welcome to do with it — lives on the Brand page.