Skip to main content

Web Crawler Quick Start

This guide gets you from zero to a running web crawl in under 5 minutes.

Built-in baseline, not a hard limit

The source scope and examples in this guide represent built-in support in Norconex Crawler v4. They are practical defaults, not a fixed ceiling. Teams can extend crawler behavior with custom connectors, parser logic, and pipeline components for customer-specific requirements.

Windows users

Replace crawl-web.sh with crawl-web.bat and ./ with .\ in all commands below.

Step 1 — Create a config file​

Create my-web-crawl.yaml with the following minimal configuration:

id: my-first-crawl
numThreads: 5
maxDepth: 3
startReferences:
  - https://example.com
urlScopeResolver:
  includeSubdomains: false
  stayOnDomain: true
committers:
  - class: LogCommitter
    ignoreContent: true
    logLevel: INFO

This crawl will:

  • Start at https://example.com
  • Follow links up to 3 levels deep
  • Log each document it processes (LogCommitter is built-in and ideal for testing)

Step 2 — Start the crawl​

./crawl-web.sh start -config=my-web-crawl.yaml

You'll see log output as pages are fetched, filtered, and committed.

Docker alternative​

If you prefer running with Docker, mount your config and logs, then run:

docker run --rm \
  -v "${PWD}:/opt/norconex/crawler/configs" \
  -v "${PWD}/logs:/opt/norconex/crawler/logs" \
  -e COLLECTOR_CONFIG_FILE=my-web-crawl.yaml \
  norconex/crawler-web:latest
During the v4 beta

No latest tag is published until 4.0.0 ships. Use the current pre-release tag instead (for example norconex/crawler-web:4.0.0-beta-1); the download page always shows it.

For Docker Compose examples and release-tag guidance, see Docker.

Step 3 — Stop and resume​

Stop the crawl at any time:

./crawl-web.sh stop -config=my-web-crawl.yaml

Norconex saves its state to disk automatically. Resume exactly where you left off by running the same start command again:

./crawl-web.sh start -config=my-web-crawl.yaml

To clear the crawler state before your next run, you can issue this command:

./crawl-web.sh clean -config=my-web-crawl.yaml

Alternatively, you can combine "clean" with the start command:

./crawl-web.sh start -clean -config=my-web-crawl.yaml

Step 4 — Send to a real target​

Replace the LogCommitter with your actual destination.

Elasticsearch:

committers:
  - class: ElasticsearchCommitter
    nodes: http://localhost:9200
    indexName: my-website

Apache Solr:

committers:
  - class: SolrCommitter
    solrURL: http://localhost:8983/solr/my-core

See the Integrations page for all available committers and their configuration.

When using ZIP distributions, external committers are downloaded separately as nx-committer-<name>-<version>.zip and their lib/*.jar files must be copied into the crawler lib/ directory. Built-in committers such as LogCommitter do not require this extra step.

CLI reference​

CommandDescription
start -config=<file>Start or resume a crawl
stop -config=<file>Gracefully stop a running crawl
clean -config=<file>Delete crawl state (forces a full recrawl)

Next steps​