Everything the crawler does, on one page

Why teams pick it​

• Scales across a clusterRun a single crawl across many nodes. Nodes find each other, share the queue and crawl state, and keep going when one drops out.
• Survives interruptionAfter a crash, a deliberate stop, or a lost node, a crawl resumes exactly where it left off instead of starting over.
• Configure without writing codeXML, JSON, or YAML — your choice — with conditions and branching expressed in the configuration itself.
• Render only when you need toPlain HTTP where speed matters, Playwright or WebDriver where a page will not give up its content without a real browser.
• Extend or embed anythingNearly every component can be swapped for your own. Run it from the command line, or embed the same engine in a Java application.
• Open source, commercially backedApache License 2.0, developed in the open. Community support on GitHub, paid support and services from Norconex.

Capabilities​

Almost everything here works the same whichever crawler you run. Where it does not, the entry is marked web-only or file-system-only.

What it can read​

Parsing is handled by the Norconex Importer, which builds on Apache Tika. The list below is representative rather than complete.

• Text and markupPlain text, HTML, XML, XFDL
• PDFPDF, including XFA dynamic forms
• Microsoft OfficeWord, Excel, PowerPoint, Visio, Outlook, Publisher, Works, Access
• OpenDocumentWriter, Calc, Impress, Draw
• Apple iWorkPages, Numbers, Keynote
• Other office suitesWordPerfect, QuattroPro, RTF, EPUB
• Email and messagingEML, MSG, MBOX, WinHelp (CHM)
• FeedsRSS, Atom, IPTC ANPA newswire
• Archiveszip, tar, gzip, bzip2, 7z, ar, cpio, arj, lzma, snappy, jar, XZ
• ImagesJPEG, PNG, GIF, TIFF, BMP, PSD, XMP, BPG, JBIG2, DJVU — metadata, plus text via OCR
• Audio and videoMP3, WAV, AIFF, AU, MP4, Flash — metadata
• Source codeJava, Groovy, C++, and others taken as-is
• Scientific dataHDF, NetCDF, GCMD DIF, ISO 19139 geographic
• Fonts and executablesTrueType, AFM, PE (exe/dll), ELF, Java class files
• Calendars and datasetsiCal, vCalendar, Stata DTA, DBF, Endnote
• OtherCBOR, WebVTT, ISO disc images, AutoCAD metadata

Feeding AI, RAG, and knowledge graphs​

A retrieval system is only as good as the text you put into it, and getting that text is most of the work. The crawler owns that part: reaching the content wherever it lives, turning it into clean text, keeping the fields that make filtering and citation possible, and coming back so what you have indexed still matches reality.

• Clean text, not markup soupText and metadata pulled from 100+ formats, embedded files extracted and processed on their own, OCR for words locked inside scans and images, and a CSS selector for when only part of a page is worth keeping.
• Split before you embedPDF pages, DOM elements, CSV rows, and XML records can each become a document in their own right, with its own reference and its own fields — so what reaches your store is already close to the unit you want to retrieve.
• Metadata that survives the tripNo fixed schema. Source URL or path, title, language, dates, content type, and anything you compute travel with the text, so retrieval can filter and answers can cite. File system sources carry owner and permission fields too.
• An index that does not driftIncremental crawls skip unchanged content, checksums catch what genuinely changed, and deleted or orphaned documents are reported — so they can leave your vector store as cleanly as they leave your search index.

Where it lands is your call. Elasticsearch, OpenSearch, and Solr all do vector search themselves now, so the same crawl that fills a keyword index fills a hybrid one. Neo4j turns documents and the links between them into a graph. Kafka hands each document to whatever computes your embeddings, and SQL/JDBC stages content in Postgres for a job to pick up. For a dedicated vector database — Qdrant, Weaviate, Pinecone, Milvus, Chroma — a committer is one method you implement. And if you would rather embed during the crawl, a script handler or an external program can call your model service and store what comes back as just another field.