Package com.norconex.crawler.fs.fetch.impl.archive

Class ArchiveFetcher

java.lang.Object
com.norconex.crawler.core.fetch.AbstractFetcher<ArchiveFetcherConfig>
com.norconex.crawler.fs.fetch.impl.AbstractNioFetcher<ArchiveFetcherConfig>
com.norconex.crawler.fs.fetch.impl.archive.ArchiveFetcher
All Implemented Interfaces:
Configurable<ArchiveFetcherConfig>, EventListener<Event>, Fetcher, EventListener, Consumer<Event>
public class ArchiveFetcher extends AbstractNioFetcher<ArchiveFetcherConfig>

Fetcher for archive file systems treated as virtual directory trees. Each entry in the archive is crawled as an individual document, enabling recursive traversal and per-document metadata extraction.

This fetcher supports archives that wrap any inner file system, including remote ones such as FTP or SFTP. When credentials are required to access the inner file system (e.g., an FTP server hosting a ZIP file), configure them via the ArchiveFetcherConfig — they are forwarded automatically to the inner scheme.

Supported archive schemes (outer layer)

  • zip:// — ZIP archives
  • jar:// — JAR files
  • tar:// — TAR archives
  • tgz:// — Gzip-compressed TAR (alias for tar:gz://)
  • tbz2:// — Bzip2-compressed TAR (alias for tar:bz2://)
  • gz:// / gzip:// — Gzip-compressed single files
  • bz2:// / bzip2:// — Bzip2-compressed single files

Example URIs

  • Local: zip:file:///backups/data.zip!/
  • FTP: zip:ftp://host/backups/data.zip!/
  • SFTP: tgz:sftp://host/exports/dump.tar.gz!/
  • HTTP: zip:https://example.com/release.zip!/

ZIP/JAR archives are mounted lazily via the JDK's built-in zipfs provider (remote sources are first spooled to a local temporary file, since zipfs requires true random access that remote NIO.2 providers such as FTP/SFTP do not support). TAR/GZIP/BZIP2 archives have no such built-in provider and are instead fully extracted once to a local temporary directory on first access, which is then treated like any other local directory.