One crawler for web and file content
Same engine, same configuration model, same operational story — whatever you point it at.
- Points at anythingWebsites, SharePoint, Alfresco and other CMIS repositories, network shares, object storage, FTP, SFTP, and WebDAV.
- Handles the awkward partsBrowser rendering for JavaScript-heavy sites, plus authenticated areas and logged-in journeys.
- Shapes content before it landsParse, filter, enrich, split, and transform content and metadata on the way through — clean text and fields, ready to index or embed.
- Publishes anywhereSearch engines, databases, graph stores, and queues out of the box — or write a committer for a vector store of your own. No lock-in.
- Extends or embeds in JavaRun it from the command line, or drop the same engine straight into your own application.
- Resumes and re-crawls cleanlyPick up after interruption, skip unchanged content, hook events, and diagnose failures.
How it works
Every document — web page or file — moves through the same three stages.
Crawl
Discover and fetch from sites, shares, and cloud storage.
Process
Parse, filter, enrich, and transform content and metadata.
Commit
Deliver to search engines, databases, and queues.
New with v4
Visual Configurator
Create, edit, and validate crawler configurations visually — without touching a text file. Upload an existing config, tweak it in the UI, then export it as XML, YAML, or JSON.
- Import and export XML, YAML, and JSON interchangeably
- Quick links to full configuration reference and live examples
- Upgrading from v3? Paste your old config, get clean v4 out
- Real-time validation and error highlighting
From zero to crawling in minutes
A minimal config is all it takes. Same shape either way — only the source and the launcher change.
Web crawler
id: acme-website
startReferences:
- https://www.example.com
numThreads: 10
maxDepth: 5
urlScopeResolver:
stayOnDomain: true
committers:
- class: ElasticsearchCommitter
indexName: web-content
nodes:
- http://localhost:9200
Save as my-crawl.yaml, then run: crawl-web.sh start -config=my-crawl.yaml
File system crawler
id: acme-documents
startReferences:
- /mnt/data/documents
referenceFilters:
- class: GenericReferenceFilter
valueMatcher:
pattern: '".*\.(pdf|docx|xlsx|txt)$"'
method: REGEX
committers:
- class: SolrCommitter
solrURL: http://localhost:8983/solr/documents
Save as my-crawl.yaml, then run: crawl-fs.sh start -config=my-crawl.yaml
What users say on GitHub
Click any quote to read it in context.
Send your data anywhere
Search platforms, databases, message queues, graph stores — and the pipelines that feed your AI.