Neo4jCommitter

Commit documents to a Neo4j graph database.

committer/neo4jJavadoc: Neo4jCommitter

Notes

Commits crawled documents to a Neo4j graph database using configurable Cypher queries for upserts and deletes. Field mappings, multi-value joining, and optional parameters for missing fields are supported.

Examples

Creates or updates one node per crawled document, keyed on its reference.

YAML

committer:
  class: Neo4jCommitter
  uri: bolt://localhost:7687
  credentials:
    username: neo4j
    password: changeme
  database: neo4j
  nodeIdProperty: id
  nodeContentProperty: content
  upsertCypher: "MERGE (d:Document {id: $id}) SET d.content = $content, d.title = $title"

JSON

{
  "committer": {
    "class": "Neo4jCommitter",
    "uri": "bolt://localhost:7687",
    "credentials": {
      "username": "neo4j",
      "password": "changeme"
    },
    "database": "neo4j",
    "nodeIdProperty": "id",
    "nodeContentProperty": "content",
    "upsertCypher": "MERGE (d:Document {id: $id}) SET d.content = $content, d.title = $title"
  }
}

XML

<committer class="Neo4jCommitter">
  <uri>bolt://localhost:7687</uri>
  <credentials>
    <username>neo4j</username>
    <password>changeme</password>
  </credentials>
  <database>neo4j</database>
  <nodeIdProperty>id</nodeIdProperty>
  <nodeContentProperty>content</nodeContentProperty>
  <upsertCypher>MERGE (d:Document {id: $id}) SET d.content = $content, d.title = $title</upsertCypher>
</committer>

Usage

Full configuration skeleton, with every option and its default

YAML

class: Neo4jCommitter
credentials:
  password: string
  passwordKey:
    size: 0
    source: KEY
    value: string
  username: string
database: string
deleteCypher: string
fieldMappings: {}
multiValuesJoiner: string
nodeContentProperty: string
nodeIdProperty: string
optionalParameters:
  - string
queue:
  class: FsQueue
  batchSize: 0
  commitLeftoversOnInit: false
  maxPerFolder: 0
  onCommitFailure:
    ignoreErrors: false
    maxRetries: 0
    retryDelay: 0
    splitBatch: OFF
restrictions:
  - fieldMatcher: {}
    valueMatcher: {}
upsertCypher: string
uri: string

JSON

{
  "class": "Neo4jCommitter",
  "credentials": {
    "password": "string",
    "passwordKey": {
      "size": 0,
      "source": "KEY",
      "value": "string"
    },
    "username": "string"
  },
  "database": "string",
  "deleteCypher": "string",
  "fieldMappings": {},
  "multiValuesJoiner": "string",
  "nodeContentProperty": "string",
  "nodeIdProperty": "string",
  "optionalParameters": [
    "string"
  ],
  "queue": {
    "class": "FsQueue",
    "batchSize": 0,
    "commitLeftoversOnInit": false,
    "maxPerFolder": 0,
    "onCommitFailure": {
      "ignoreErrors": false,
      "maxRetries": 0,
      "retryDelay": 0,
      "splitBatch": "OFF"
    }
  },
  "restrictions": [
    {
      "fieldMatcher": {},
      "valueMatcher": {}
    }
  ],
  "upsertCypher": "string",
  "uri": "string"
}

XML

<neo4jCommitter>
  <class>Neo4jCommitter</class>
  <credentials>
    <password>string</password>
    <passwordKey>
      <size>0</size>
      <source>KEY</source>
      <value>string</value>
    </passwordKey>
    <username>string</username>
  </credentials>
  <database>string</database>
  <deleteCypher>string</deleteCypher>
  <fieldMappings/>
  <multiValuesJoiner>string</multiValuesJoiner>
  <nodeContentProperty>string</nodeContentProperty>
  <nodeIdProperty>string</nodeIdProperty>
  <optionalParameters>
    <optionalParameter>string</optionalParameter>
  </optionalParameters>
  <queue>
    <class>FsQueue</class>
    <batchSize>0</batchSize>
    <commitLeftoversOnInit>false</commitLeftoversOnInit>
    <maxPerFolder>0</maxPerFolder>
    <onCommitFailure/>
  </queue>
  <restrictions>
    <restriction>
      <fieldMatcher/>
      <valueMatcher/>
    </restriction>
  </restrictions>
  <upsertCypher>string</upsertCypher>
  <uri>string</uri>
</neo4jCommitter>

Properties

Property	Type	Required	Default
credentials	Credentials	No	Credentials
database	string	No	-
deleteCypher	string	No	-
fieldMappings	Record<string, string>	No	-
multiValuesJoiner	string	No	-
nodeContentProperty	string	No	content
nodeIdProperty	string	No	id
optionalParameters	string[]	No	-
queue	CommitterQueue	No	FsQueue
restrictions	PropertyMatcher[]	No	-
upsertCypher	string	No	-
uri	string	No	-

Property Details

credentials

Username and password to Neo4j.

database

Database name when using one other than the default.

deleteCypher

Cypher query for deleting relationships in Neo4j. Typically, you want to use "nodeIdProperty" value to delete nodes having a matching property value. The query parameters correspond the document fields.

fieldMappings

Mapping of source fields to target ones (Neo4j property).

multiValuesJoiner

Characters used to join a multi-valued field into the single value stored on the node. Default is |. Choose something that cannot appear in your values if you intend to split them apart again later.

nodeContentProperty

Property name where to store the document content in Neo4j graph entries. Use it as a cypher parameter in your configured "upsertCypher" query. Default is "content".

nodeIdProperty

Property name where to store the document reference in Neo4j graph entries. Use it as a cypher parameter to uniquely identify your graph entries in your configured "upsertCypher" and "deleteCypher" queries. Default is "id".

optionalParameters

Comma-separated list of parameter names that can be missing when creating the query. They will be set to NullValue to avoid client exception for missing parameters.

queue

Configures the committer queue used to batch documents before sending them to the target system. Adjust queue settings to tune throughput and memory usage.

restrictions

Defines filter conditions that restrict which documents this committer processes. Documents not matching the restrictions are ignored by this committer.

upsertCypher

Cypher query for adding relationships to Neo4j. Typically, you want to use MERGE and the "nodeIdProperty" value to update existing entries matching the ID. In order to later delete all nodes related to an entry, make sure to add the ID property on all appropriate nodes. The query parameters correspond the document fields.

uri

Required connection URI. E.g., bolt://localhost:7687.