M365GraphFetcher

Microsoft 365 Graph fetcher for SharePoint Online and OneDrive for Business.

Notes

Fetches documents from Microsoft Graph using stable item-ID-based references for SharePoint and OneDrive.

Supported schemes:

m365sp://{tenant}/sites/{siteId}
m365sp://{tenant}/siteurl?url={url-encoded-absolute-site-url}
m365sp://{tenant}/sites/{siteId}/drives/{driveId}/items/{itemId}
m365od://{tenant}/users/{userId}
m365od://{tenant}/users/{userId}/drives/{driveId}/items/{itemId}

V1 authentication is app-only (client credentials).

Discovery entry references (/sites/{siteId}, /siteurl?..., and /users/{userId}) are supported and expand to drive roots before normal item traversal.

Retry model:

Framework retry is controlled by crawler-level fetchersMaxRetries and fetchersRetryDelay.
Optional M365-native retry can be enabled for throttling/transient Graph statuses (for example 429), including Retry-After support.
Keep native retry disabled to rely solely on framework retry for consistency.

Incremental source-delta mode:

When crawler-level changeDiscovery is set to SOURCE_DELTA and the crawl runs in incremental mode, drive references use Microsoft Graph delta APIs instead of full child enumeration.
sourceDeltaExpansion controls how non-drive start references behave in SOURCE_DELTA mode.
- SELF_ONLY: the start reference must already be a drive boundary.
- INCLUDE_CHILD_DRIVES: site, site URL, and user start references are expanded to child drives, with one persisted delta cursor per drive.
Delta cursors are persisted per drive reference for the current crawler session and reused on the next incremental run.
If Microsoft Graph rejects a stored cursor (400, 404, or 410), the fetcher automatically clears it and restarts delta discovery from a fresh drive delta request.
Deletions are handled from explicit Graph delta tombstones. Missing items are not inferred from absence while changeDiscovery=SOURCE_DELTA.
If a configured or discovered drive boundary itself becomes NOT_FOUND, the crawler re-queues known descendants from the previous baseline for deletion processing, so committers receive deletion requests for previously indexed items under that drive.

Examples

The following example starts from a known SharePoint drive item.

YAML
JSON
XML

class: M365GraphFetcher
tenantId: contoso.onmicrosoft.com
clientId: 11111111-2222-3333-4444-555555555555
clientSecret: <secret>
validStatusCodes:
  - 200
  - 204
notFoundStatusCodes:
  - 404
  - 410

{
  "class": "M365GraphFetcher",
  "tenantId": "contoso.onmicrosoft.com",
  "clientId": "11111111-2222-3333-4444-555555555555",
  "clientSecret": "<secret>",
  "validStatusCodes": [
    200,
    204
  ],
  "notFoundStatusCodes": [
    404,
    410
  ]
}

<class>M365GraphFetcher</class>
<tenantId>contoso.onmicrosoft.com</tenantId>
<clientId>11111111-2222-3333-4444-555555555555</clientId>
<clientSecret>&lt;secret&gt;</clientSecret>
<validStatusCodes>200</validStatusCodes>
<validStatusCodes>204</validStatusCodes>
<notFoundStatusCodes>404</notFoundStatusCodes>
<notFoundStatusCodes>410</notFoundStatusCodes>

Use a start reference such as:

m365sp://contoso.onmicrosoft.com/sites/{siteId}/drives/{driveId}/items/{itemId}

For discovery bootstrap, start from a site or site URL instead:

m365sp://contoso.onmicrosoft.com/sites/{siteId}
m365sp://contoso.onmicrosoft.com/siteurl?url=https%3A%2F%2Fcontoso.sharepoint.com%2Fsites%2Fengineering

The following example starts from a known OneDrive drive item.

YAML
JSON
XML

class: M365GraphFetcher
tenantId: contoso.onmicrosoft.com
clientId: 11111111-2222-3333-4444-555555555555
clientSecret: <secret>
nativeRetryEnabled: true
nativeRetryMaxRetries: 4
nativeRetryStatusCodes:
  - 429
  - 503
  - 504

{
  "class": "M365GraphFetcher",
  "tenantId": "contoso.onmicrosoft.com",
  "clientId": "11111111-2222-3333-4444-555555555555",
  "clientSecret": "<secret>",
  "nativeRetryEnabled": true,
  "nativeRetryMaxRetries": 4,
  "nativeRetryStatusCodes": [
    429,
    503,
    504
  ]
}

<class>M365GraphFetcher</class>
<tenantId>contoso.onmicrosoft.com</tenantId>
<clientId>11111111-2222-3333-4444-555555555555</clientId>
<clientSecret>&lt;secret&gt;</clientSecret>
<nativeRetryEnabled>true</nativeRetryEnabled>
<nativeRetryMaxRetries>4</nativeRetryMaxRetries>
<nativeRetryStatusCodes>429</nativeRetryStatusCodes>
<nativeRetryStatusCodes>503</nativeRetryStatusCodes>
<nativeRetryStatusCodes>504</nativeRetryStatusCodes>

Use a start reference such as:

m365od://contoso.onmicrosoft.com/users/{userId}/drives/{driveId}/items/{itemId}

For discovery bootstrap from the user boundary, start with:

m365od://contoso.onmicrosoft.com/users/{userId}

The following example is suited for incremental delta crawling from a drive boundary.

YAML
JSON
XML

class: M365GraphFetcher
tenantId: contoso.onmicrosoft.com
clientId: 11111111-2222-3333-4444-555555555555
clientSecret: <secret>
nativeRetryEnabled: true
nativeRetryStatusCodes:
  - 429
  - 503
  - 504

{
  "class": "M365GraphFetcher",
  "tenantId": "contoso.onmicrosoft.com",
  "clientId": "11111111-2222-3333-4444-555555555555",
  "clientSecret": "<secret>",
  "nativeRetryEnabled": true,
  "nativeRetryStatusCodes": [
    429,
    503,
    504
  ]
}

<class>M365GraphFetcher</class>
<tenantId>contoso.onmicrosoft.com</tenantId>
<clientId>11111111-2222-3333-4444-555555555555</clientId>
<clientSecret>&lt;secret&gt;</clientSecret>
<nativeRetryEnabled>true</nativeRetryEnabled>
<nativeRetryStatusCodes>429</nativeRetryStatusCodes>
<nativeRetryStatusCodes>503</nativeRetryStatusCodes>
<nativeRetryStatusCodes>504</nativeRetryStatusCodes>

Use a start reference such as:

m365od://contoso.onmicrosoft.com/users/{userId}/drives/{driveId}

Then configure the crawler itself with:

crawlMode: INCREMENTAL
changeDiscovery: SOURCE_DELTA

If you want to keep site or user start references as the user-facing scope, enable child-drive expansion:

YAML
JSON
XML

class: M365GraphFetcher
tenantId: contoso.onmicrosoft.com
clientId: 11111111-2222-3333-4444-555555555555
clientSecret: <secret>
sourceDeltaExpansion: INCLUDE_CHILD_DRIVES

{
  "class": "M365GraphFetcher",
  "tenantId": "contoso.onmicrosoft.com",
  "clientId": "11111111-2222-3333-4444-555555555555",
  "clientSecret": "<secret>",
  "sourceDeltaExpansion": "INCLUDE_CHILD_DRIVES"
}

<class>M365GraphFetcher</class>
<tenantId>contoso.onmicrosoft.com</tenantId>
<clientId>11111111-2222-3333-4444-555555555555</clientId>
<clientSecret>&lt;secret&gt;</clientSecret>
<sourceDeltaExpansion>INCLUDE_CHILD_DRIVES</sourceDeltaExpansion>

Valid SOURCE_DELTA start references with that setting include:

m365sp://contoso.onmicrosoft.com/sites/{siteId}
m365sp://contoso.onmicrosoft.com/siteurl?url=https%3A%2F%2Fcontoso.sharepoint.com%2Fsites%2Fengineering
m365od://contoso.onmicrosoft.com/users/{userId}

Usage

Full configuration skeleton, with every option and its default

YAML
JSON
XML

class: M365GraphFetcher
authorityHost: string
clientId: string
clientSecret: string
graphBaseUrl: string
nativeRetryBaseDelay: 0
nativeRetryEnabled: false
nativeRetryMaxRetries: 0
nativeRetryStatusCodes:
  - 0
notFoundStatusCodes:
  - 0
referenceFilters: []
sourceDeltaExpansion: SELF_ONLY
tenantId: string
validStatusCodes:
  - 0

{
  "class": "M365GraphFetcher",
  "authorityHost": "string",
  "clientId": "string",
  "clientSecret": "string",
  "graphBaseUrl": "string",
  "nativeRetryBaseDelay": 0,
  "nativeRetryEnabled": false,
  "nativeRetryMaxRetries": 0,
  "nativeRetryStatusCodes": [
    0
  ],
  "notFoundStatusCodes": [
    0
  ],
  "referenceFilters": [],
  "sourceDeltaExpansion": "SELF_ONLY",
  "tenantId": "string",
  "validStatusCodes": [
    0
  ]
}

<m365GraphFetcher>
  <class>M365GraphFetcher</class>
  <authorityHost>string</authorityHost>
  <clientId>string</clientId>
  <clientSecret>string</clientSecret>
  <graphBaseUrl>string</graphBaseUrl>
  <nativeRetryBaseDelay>0</nativeRetryBaseDelay>
  <nativeRetryEnabled>false</nativeRetryEnabled>
  <nativeRetryMaxRetries>0</nativeRetryMaxRetries>
  <nativeRetryStatusCodes>
    <nativeRetryStatusCode>0</nativeRetryStatusCode>
  </nativeRetryStatusCodes>
  <notFoundStatusCodes>
    <notFoundStatusCode>0</notFoundStatusCode>
  </notFoundStatusCodes>
  <referenceFilters>
    <referenceFilter/>
  </referenceFilters>
  <sourceDeltaExpansion>SELF_ONLY</sourceDeltaExpansion>
  <tenantId>string</tenantId>
  <validStatusCodes>
    <validStatusCode>0</validStatusCode>
  </validStatusCodes>
</m365GraphFetcher>

Properties

Property	Type	Required	Default
`authorityHost`	`string`	No	`https://login.microsoftonline.com`
`clientId`	`string`	No	-
`clientSecret`	`string`	No	-
`graphBaseUrl`	`string`	No	`https://graph.microsoft.com/v1.0`
`nativeRetryBaseDelay`	`integer`	No	`1000`
`nativeRetryEnabled`	`boolean`	No	`false`
`nativeRetryMaxRetries`	`integer`	No	`3`
`nativeRetryStatusCodes`	`integer`[]	No	[`429`, `503`, `504`]
`notFoundStatusCodes`	`integer`[]	No	[`404`]
`referenceFilters`	`ReferenceFilter`[]	No	-
`sourceDeltaExpansion`	`enum`	No	`SELF_ONLY`
`tenantId`	`string`	No	-
`validStatusCodes`	`integer`[]	No	[`200`, `201`, `202`, `204`]

Property Details

`authorityHost`

Authentication authority host. Defaults to https://login.microsoftonline.com.

`clientId`

Azure AD application (client) ID.

`clientSecret`

Azure AD client secret used for client-credentials token acquisition.

`graphBaseUrl`

Microsoft Graph API root URL. Defaults to https://graph.microsoft.com/v1.0.

`nativeRetryBaseDelay`

Base delay for native retries when Retry-After is absent. Default is PT1S.

`nativeRetryEnabled`

When true, enables M365-native retry for throttling/transient statuses. Default is false.

`nativeRetryMaxRetries`

Maximum native retries when nativeRetryEnabled is true. Default is 3.

`nativeRetryStatusCodes`

Status codes eligible for native retry when enabled. Defaults to 429, 503, and 504.

`notFoundStatusCodes`

HTTP status codes treated as "not found" outcomes. Defaults to 404.

`referenceFilters`

Filters applied to document references (URLs or file paths) to control which ones are fetched. Documents whose references do not pass the filters are skipped.

`sourceDeltaExpansion`

How SOURCE_DELTA start references are expanded into Graph delta roots. Defaults to SELF_ONLY.

SELF_ONLY: only drive start references are valid delta roots.
INCLUDE_CHILD_DRIVES: site, site URL, and user references expand to child drives and keep one cursor per drive.

Allowed Values

SELF_ONLY
INCLUDE_CHILD_DRIVES

`tenantId`

Tenant ID used for token acquisition and tenant boundary validation.

`validStatusCodes`

HTTP status codes treated as successful Graph responses. Defaults to 200, 201, 202, and 204.

Give feedback:Discuss this page|Report an issue

M365GraphFetcher

Notes​

Examples​

Usage​

Properties​

Property Details​

authorityHost​

clientId​

clientSecret​

graphBaseUrl​

nativeRetryBaseDelay​

nativeRetryEnabled​

nativeRetryMaxRetries​

nativeRetryStatusCodes​

notFoundStatusCodes​

referenceFilters​

sourceDeltaExpansion​

Allowed Values​

tenantId​

validStatusCodes​

Notes

Examples

Usage

Properties

Property Details

`authorityHost`

`clientId`

`clientSecret`

`graphBaseUrl`

`nativeRetryBaseDelay`

`nativeRetryEnabled`

`nativeRetryMaxRetries`

`nativeRetryStatusCodes`

`notFoundStatusCodes`

`referenceFilters`

`sourceDeltaExpansion`

Allowed Values

`tenantId`

`validStatusCodes`