Welcome to Norconex Crawler
Norconex Crawler is an open-source, enterprise-grade crawling framework that collects content from websites and file systems and delivers it to search engines, databases, and data pipelines — without vendor lock-in.
Two crawlers, one framework
| Web Crawler | File system Crawler | |
|---|---|---|
| Sources | HTTP/S websites and web applications | Local and remote file storage across many protocols |
| How it navigates | Follows hyperlinks; can render JavaScript | Traverses directory trees; resolves paths by protocol |
| Typical use case | Web indexing, site monitoring, content archiving | Document management, network storage indexing |
Both crawlers share the same configuration model, pipeline, and committer system. Learn one, know both.
The source types highlighted in this guide represent built-in support in Norconex Crawler v4. They are practical defaults, not a fixed ceiling. Teams can extend crawler behavior with custom connectors, parser logic, and pipeline components for customer-specific requirements.
Core concepts at a glance
- Crawl session — one complete pass over everything the crawler is configured to reach
- Crawl run — one execution of the crawler; an interrupted session resumes across several runs
- Incremental crawling — from the second session onward, only what changed is committed
- Pipeline — every document flows through: Crawl → Process → Commit
- Committers — pluggable output targets (Elasticsearch, Solr, Kafka, SQL, Neo4j, ...)
The Visual Configurator, a web application Norconex hosts and offers free of charge, lets you build and validate crawler configs visually. The Configurator itself is not open source — the crawler is, and remains so.
Where to go next
- New to Norconex Crawler? → Installation then Docker, Web Quick Start, File System Quick Start, and FS Fetchers Quickstart
- Embedding in Java? → Java Integration
- Need config rules and defaults? → Configuration Semantics
- How does it all work? → Concepts
- Upgrading from v3? → Migration Guide