Skip to main content

Content sources

Crawl the systems you already run

One engine reaches websites, network shares, cloud storage, and the document management systems your content actually lives in. Where there is no purpose-built connector, there is almost always a way in.

  • 17Source types
  • 16Reached by their own protocol
  • AnySite the web crawler can reach

Document and content management

The systems your organisation files things in, reached through their own APIs.

  • Microsoft 365 — SharePoint and OneDriveSharePoint sites and OneDrive accounts through the Microsoft Graph API.m365sp://m365od://File system crawler
  • CMIS repositoriesAny repository exposing the CMIS standard over its Atom binding — Alfresco, Nuxeo, OpenText, Documentum, IBM FileNet, and others.cmis://File system crawler
  • Google DriveGoogle Drive files and shared drives.gdrive://File system crawler
  • BoxBox folders and their contents.box://File system crawler
  • EgnyteEgnyte cloud file storage.egnyte://File system crawler

Repositories reached over CMIS

CMIS is an OASIS standard, so the crawler talks to these through the same fetcher and the same configuration. Check your product's docs for its CMIS endpoint — it is usually present even when the vendor does not advertise it.

  • Alfresco Content ServicesCMIS 1.0/1.1 over Atom
  • NuxeoAtom and Browser bindings
  • OpenText Content Servervia its CMIS interface
  • OpenText Documentumvia its CMIS layer
  • IBM FileNetvia its CMIS layer
  • Any CMIS-compliant repositorythe standard, not a list

Cloud object storage

Buckets and data lakes, crawled in place.

  • Amazon S3S3 buckets, and S3-compatible stores such as MinIO and Wasabi.s3://File system crawler
  • Azure Blob StorageAzure Blob containers.azblob://azureblob://File system crawler
  • Azure Data Lake Storage Gen2ADLS Gen2 file systems, over the ABFS driver.abfs://abfss://File system crawler
  • Google Cloud StorageGoogle Cloud Storage buckets.gs://File system crawler
  • HDFSHadoop file systems through the WebHDFS REST interface.webhdfs://File system crawler

Servers, shares and disks

The protocols that have been carrying documents for decades.

  • Local disks and mounted volumesAnything the host can see, including network drives already mounted by the operating system.file://absolute pathsFile system crawler
  • SMB / CIFS sharesWindows and Samba network shares, with their ACL metadata.smb://File system crawler
  • FTP and FTPSFTP servers, plain or over TLS.ftp://ftps://File system crawler
  • SFTPFile transfer over SSH.sftp://File system crawler
  • WebDAVWebDAV servers, plain or over TLS.webdav://webdavs://File system crawler
  • ArchivesArchives walked as if they were folders, including ones sitting on a remote file system.zip://jar://tar://tgz://tbz2://gz://bz2://File system crawler

The open web

Anything with a URL, whether or not it has an API.

  • WebsitesAny site the crawler can reach, including authenticated areas and pages that only render under a real browser.http://https://Web crawler

Not on the list?

Three routes cover almost everything else.

  • It speaks CMISMost enterprise content repositories expose the CMIS standard, whether or not they advertise it. If yours does, it is already supported — no product-specific connector required.
  • It has a web interfaceAnything served over HTTP can be crawled as a website, including behind a login. This is the usual answer for intranets, wikis, and systems whose API is not worth the trouble.
  • It can exportSystems that can dump content to a share or a bucket become a file system crawl. Less elegant than an API, and it works everywhere.

Still stuck? Fetchers are an extension point like everything else — write your own, or ask us.

How do I configure one?

Each fetcher has its own options — credentials, regions, timeouts, and the rest — in the configuration reference.

Open the reference

Where does it all go?

Crawled content is delivered by a committer — search engines, databases, queues, or a target you write yourself.

See the destinations