Skip to main content

Installation

Requirements​

  • Java 21 or later (JRE is sufficient for CLI use)

Verify your Java version:

java -version

Standard vs Full distributions​

Each crawler ships in two variants:

  • Standard — covers most use cases and has a smaller footprint.
  • Full — includes everything in Standard plus optional extras: OCR, NLP language detection, extra parsers, scripting, and more.

Start with Standard unless you know you need those extras.

Option 1 — Download the ZIP​

  1. Go to the Download page and grab the latest release ZIP for your chosen crawler.

  2. Extract it (the folder name matches the ZIP name):

    # Standard distribution
    unzip nx-crawler-web-4.0.0-standard.zip

    # Full distribution
    unzip nx-crawler-web-4.0.0-full.zip

  3. The extracted folder looks like this:

    nx-crawler-web-4.0.0-standard/
      crawl-web.sh      ← Linux/macOS launch script
      crawl-web.bat     ← Windows launch script
      log4j2.xml        ← logging configuration
      examples/         ← ready-to-run sample configurations
      lib/              ← all JAR dependencies
      scripts/          ← utility scripts
      third-party/      ← third-party notices
      README.txt
      LICENSE.txt

  4. On Linux/macOS, make the launch script executable:

    chmod +x crawl-web.sh

  5. Verify the installation:

    # Linux/macOS
    ./crawl-web.sh --help

    # Windows
    .\crawl-web.bat --help
File System Crawler

All examples above use the Web Crawler (nx-crawler-web, crawl-web.sh). The File System Crawler is installed identically — substitute web with fs in every filename and command (e.g. nx-crawler-fs-4.0.0-standard.zip, crawl-fs.sh).

Run the bundled examples​

Every distribution ships an examples/ folder with two ready-to-run configurations and a HOWTO_RUN_EXAMPLES.txt. Run them from the folder you unpacked, before writing a config of your own.

  • minimum/ — the smallest configuration that does something, plus a couple of recommended settings.
  • complex/ — the same crawl with configuration variables, an included importer fragment, and a production profile that activates extra configuration sections. A complex-config.variables file supplies the defaults; pass -variables=examples/complex/production.variables to switch profile.

The Web Crawler examples crawl pages published at https://crawler.norconex.com/examples/, so they need internet access. The File System Crawler examples crawl a small set of sample files shipped inside the ZIP, so they run offline.

# Web Crawler (Linux/macOS; use crawl-web.bat and \ on Windows)
./crawl-web.sh start -config=examples/minimum/minimum-config.xml

# File System Crawler
./crawl-fs.sh start -config=examples/minimum/minimum-config.xml

What to expect:

  • Log lines as each document is fetched, imported, and committed, ending with a Total processed: summary.
  • Collected documents under examples-output/minimum/committed/ as XML files. Open one — the text the crawler extracted includes a line confirming the run worked.
  • Re-running does nothing until you change a source document or run clean first: the crawler only commits what changed since last time.

To see what the templated complex configuration resolves to without crawling, use configrender:

./crawl-web.sh configrender -config=examples/complex/complex-config.xml
./crawl-web.sh configrender -config=examples/complex/complex-config.xml \
    -variables=examples/complex/production.variables

To point an example at your own content, edit startReferences in minimum-config.xml (or set startUrl / sourceDir in a variables file for the complex example). File System Crawler start references must be an absolute path or a file:// URL — the shipped example uses ${user.dir} to resolve the bundled folder's absolute path at runtime.

Install external committers when using ZIP distributions​

ZIP distributions include built-in committer support from nx-committer-core (for example, LogCommitter).

If you want an external committer such as Elasticsearch, Solr, SQL, Kafka, Neo4j, Amazon CloudSearch, or Azure Cognitive Search, download the matching nx-committer-<name>-<version>.zip package separately and copy its JAR files into the crawler lib/ directory.

Example (Linux/macOS):

# From the crawler installation directory
unzip -j nx-committer-elasticsearch-4.0.0.zip "*/lib/*.jar" -d lib/

Example (Windows PowerShell):

# From the crawler installation directory
Expand-Archive .\nx-committer-elasticsearch-4.0.0.zip -DestinationPath .\tmp-committer
Copy-Item .\tmp-committer\*\lib\*.jar -Destination .\lib\ -Force
Remove-Item .\tmp-committer -Recurse -Force

After copying the JARs, start (or restart) the crawler normally.

Option 2 — Use Docker images​

Norconex publishes official images to both Docker registries:

  • Docker Hub: norconex/crawler-web, norconex/crawler-fs, norconex/crawler-web-playwright
  • GitHub Container Registry: ghcr.io/norconex/crawler-web, ghcr.io/norconex/crawler-fs, ghcr.io/norconex/crawler-web-playwright

Registry pages:

Example pulls:

docker pull norconex/crawler-web:latest
docker pull ghcr.io/norconex/crawler-fs:latest
During the v4 beta

Version 4.0.0 has not been released yet, so there is no latest tag published yet. Until it ships, use the current pre-release tag instead (for example norconex/crawler-web:4.0.0-beta-1) — the download page always shows the current one.

Official Docker images already bundle the external committer JARs, so no separate committer installation is required in containers.

For Docker Compose examples, tag strategy (latest vs edge), and runtime tips, see Docker.

Option 3 — Maven dependency​

For embedding the crawler in a Java application, see Java Integration.

Configuration rules and defaults​

Before authoring larger configs, read Configuration Semantics for shared behavior such as defaults, null vs empty values, variable resolution, and reusable fragments.

Option 4 — Build from source​

git clone https://github.com/Norconex/crawler.git
cd crawlers

# Web crawler
mvn clean package -pl crawler/web -Dmaven.test.skip=true

# File system crawler
mvn clean package -pl crawler/fs -Dmaven.test.skip=true

The distribution ZIP files will be in crawler/web/target/ or crawler/fs/target/.

Next: Docker, Web Quick Start, or File System Quick Start