Serialized Form

Package com.norconex.committer.azurecognitivesearch

Class com.norconex.committer.azurecognitivesearch.AzureSearchCommitterConfig

class AzureSearchCommitterConfig extends BaseBatchCommitterConfig implements Serializable

serialVersionUID:

1L

Serialized Fields

apiKey

String apiKey

The Azure API admin key.

apiVersion

String apiVersion

The Azure API version. Default is AzureSearchCommitterConfig.DEFAULT_API_VERSION.

arrayFields

String arrayFields

The fields which values should always be treated as array. Expects a comma-separated-value list or regular expression when the returned value of AzureSearchCommitterConfig.isArrayFieldsRegex() is true.

See Also:

• AzureSearchCommitterConfig.setArrayFieldsRegex(boolean)
• AzureSearchCommitterConfig.isArrayFieldsRegex()

arrayFieldsRegex

boolean arrayFieldsRegex

Whether the list of fields to be always treated as array is represented as regular expression.

See Also:

• AzureSearchCommitterConfig.setArrayFields(String)
• AzureSearchCommitterConfig.getArrayFields()

disableDocKeyEncoding

boolean disableDocKeyEncoding

Whether to disable document reference encoding. By default, references are encoded using a URL-safe Base64 encoding. When true, document references will be sent as is if they pass validation.

endpoint

String endpoint

Gets the Azure Search endpoint. (https://[service name].search.windows.net).

ignoreResponseErrors

boolean ignoreResponseErrors

Whether to ignore response errors. By default, an exception is thrown if the Azure Search response contains an error. When true the errors are logged instead.

ignoreValidationErrors

boolean ignoreValidationErrors

Whether to ignore validation errors. By default, an exception is thrown if a document contains a field that Azure Search will reject. When true the validation errors are logged instead and the faulty field or document is not committed.

indexName

String indexName

The target index name.

proxySettings

ProxySettings proxySettings

sourceKeyField

String sourceKeyField

The document field name containing the value to be stored in Azure Search document key field. Default is not a field (null), but rather the document reference.

targetContentField

String targetContentField

The name of the Azure Search field where content will be stored. Default is "content". Specifying a null value will disable storing the content.

targetKeyField

String targetKeyField

The name of Azure Search key field where to store a document unique identifier (sourceKeyField). Default is "id".

useWindowsAuth

boolean useWindowsAuth

Whether to use integrated Windows Authentication (if applicable).

Package com.norconex.committer.core

Class com.norconex.committer.core.CommitterEvent

class CommitterEvent extends Event implements Serializable

serialVersionUID:

1L

Exception Class com.norconex.committer.core.CommitterException

class CommitterException extends Exception implements Serializable

serialVersionUID:

1L

Package com.norconex.committer.core.batch.queue

Exception Class com.norconex.committer.core.batch.queue.CommitterQueueException

class CommitterQueueException extends CommitterException implements Serializable

serialVersionUID:

1L

Package com.norconex.committer.core.service

Class com.norconex.committer.core.service.CommitterServiceEvent

class CommitterServiceEvent extends Event implements Serializable

serialVersionUID:

1L

Exception Class com.norconex.committer.core.service.CommitterServiceException

class CommitterServiceException extends RuntimeException implements Serializable

serialVersionUID:

1L

Package com.norconex.committer.googlecloudsearch

Class com.norconex.committer.googlecloudsearch.GoogleCloudSearchCommitterConfig

class GoogleCloudSearchCommitterConfig extends BaseBatchCommitterConfig implements Serializable

serialVersionUID:

1L

Serialized Fields

aclInheritance

GoogleCloudSearchCommitterConfig.AclInheritanceMapping aclInheritance

ACL inheritance configuration, mapping a metadata field holding a parent item reference to an ACL inheritance type.

aclMappings

List<GoogleCloudSearchCommitterConfig.AclMapping> aclMappings

Mappings of metadata fields to Google Cloud Search ACL principals. Cloud Search requires every indexed item to carry an ACL. When no mapping resolves to a reader for a given document (or none is configured at all) and no ACL is being inherited, the item defaults to being readable by the entire Google Workspace domain.

apiEndpoint

String apiEndpoint

Overrides the Google Cloud Search API root URL. Mainly useful for testing against a local mock server.

applicationName

String applicationName

The application name sent to Google Cloud Search. Default is "Norconex Google Cloud Search Committer".

connectorName

String connectorName

The connector name sent with indexing requests. Defaults to the application name when not set.

dataSourceId

String dataSourceId

The Google Cloud Search data source ID to index documents into.

failOnDeleteNotFound

boolean failOnDeleteNotFound

Whether a delete request targeting an item that does not exist in the index is treated as a failure. Crawlers routinely delete references that were never indexed (rejected, orphan or unmodified documents), and Google Cloud Search answers those with a "404 NOT_FOUND". Since the desired outcome (the item is absent) already holds, such responses are ignored by default. Set to true to have them fail the batch instead. Only delete requests are affected: a "404" on an index request always fails, since it signals a bad data source or connector name.

httpBackoffInitialIntervalMillis

int httpBackoffInitialIntervalMillis

Optional exponential backoff initial interval in milliseconds. Use -1 to keep Google client defaults.

httpBackoffMaxElapsedTimeMillis

int httpBackoffMaxElapsedTimeMillis

Optional exponential backoff maximum elapsed time in milliseconds. Use -1 to keep Google client defaults.

httpBackoffMaxIntervalMillis

int httpBackoffMaxIntervalMillis

Optional exponential backoff maximum interval in milliseconds. Use -1 to keep Google client defaults.

httpConnectTimeoutMillis

int httpConnectTimeoutMillis

Optional HTTP connect timeout in milliseconds applied to Google API requests. Use -1 to keep Google client defaults.

httpMaxRetries

int httpMaxRetries

Optional HTTP retry count for Google API requests. Use -1 to keep Google client defaults.

httpReadTimeoutMillis

int httpReadTimeoutMillis

Optional HTTP read timeout in milliseconds applied to Google API requests. Use -1 to keep Google client defaults.

keepSourceIdField

boolean keepSourceIdField

Whether to keep the source ID field in the document metadata after it has been extracted.

metadataMappings

List<GoogleCloudSearchCommitterConfig.MetadataMapping> metadataMappings

Metadata mappings to populate Google Cloud Search predefined metadata fields. Mapped source fields are excluded from structured data unless keepFromField is set to true. See Google reference: https://developers.google.com/workspace/cloud-search/docs/reference/rest/v1/indexing.datasources.items#ItemMetadata

requestMode

GoogleCloudSearchCommitterConfig.RequestMode requestMode

The priority/consistency mode sent with indexing and delete requests. Default is GoogleCloudSearchCommitterConfig.RequestMode.ASYNCHRONOUS, suited to content connectors performing full or incremental crawls.

secretKeyPath

String secretKeyPath

Path to the Google service account JSON key file, with access to the Cloud Search indexing API.

sourceIdField

String sourceIdField

Metadata field holding the value to use as the document unique identifier. Defaults to the document reference.

structuredDataMappings

List<GoogleCloudSearchCommitterConfig.StructuredDataMapping> structuredDataMappings

Declares the Google Cloud Search structured data value type to use for individual metadata fields sent as structured data. Fields without a matching mapping are sent as GoogleCloudSearchCommitterConfig.StructuredDataType.TEXT, which is always a safe choice since every Cloud Search schema property accepts text. Declaring another type here must match the type actually registered for that property name in the connected data source's schema; this committer has no way to verify that on its own.

uploadFormat

GoogleCloudSearchCommitterConfig.UploadFormat uploadFormat

The format used to upload document content. Default is GoogleCloudSearchCommitterConfig.UploadFormat.RAW.

Class com.norconex.committer.googlecloudsearch.GoogleCloudSearchCommitterConfig.AclInheritanceMapping

class AclInheritanceMapping extends Object implements Serializable

serialVersionUID:

1L

Serialized Fields

aclInheritanceType

GoogleCloudSearchCommitterConfig.AclInheritanceType aclInheritanceType

ACL inheritance type. Default is GoogleCloudSearchCommitterConfig.AclInheritanceType.NOT_APPLICABLE.

fromField

String fromField

Metadata field holding the parent item reference.

Class com.norconex.committer.googlecloudsearch.GoogleCloudSearchCommitterConfig.AclMapping

class AclMapping extends Object implements Serializable

serialVersionUID:

1L

Serialized Fields

fromField

String fromField

Metadata field holding the principal value(s).

principalType

GoogleCloudSearchCommitterConfig.PrincipalType principalType

Type of principal the field value represents.

target

GoogleCloudSearchCommitterConfig.AclTarget target

Which ACL list the resolved principal(s) are added to.

Class com.norconex.committer.googlecloudsearch.GoogleCloudSearchCommitterConfig.MetadataMapping

class MetadataMapping extends Object implements Serializable

serialVersionUID:

1L

Serialized Fields

defaultValue

String defaultValue

Optional fallback value when GoogleCloudSearchCommitterConfig.MetadataMapping.fromField is missing or blank.

fromField

String fromField

Optional source metadata field in crawler documents.

keepFromField

boolean keepFromField

Whether to keep GoogleCloudSearchCommitterConfig.MetadataMapping.fromField in structured data.

toField

MetadataField toField

Mandatory target Google Cloud Search metadata field.

Class com.norconex.committer.googlecloudsearch.GoogleCloudSearchCommitterConfig.StructuredDataMapping

class StructuredDataMapping extends Object implements Serializable

serialVersionUID:

1L

Serialized Fields

field

String field

Metadata/structured-data field name this mapping applies to.

type

GoogleCloudSearchCommitterConfig.StructuredDataType type

Value type to use for this field. Default is TEXT.

Package com.norconex.committer.idol

Class com.norconex.committer.idol.IdolCommitterConfig

class IdolCommitterConfig extends BaseBatchCommitterConfig implements Serializable

serialVersionUID:

1L

Serialized Fields

cfs

boolean cfs

Whether the IDOL index URL points to a Connector Framework Server (CFS).

databaseName

String databaseName

The IDOL database name.

dreAddDataParams

Map<String,String> dreAddDataParams

Gets the DREADDDATA optional URL parameters.

dreDeleteRefParams

Map<String,String> dreDeleteRefParams

The DREDELETEREF optional URL parameters.

sourceContentField

String sourceContentField

The document field name containing the value to be stored in IDOL DRECONTENT field. Set to null in order to use the document content stream instead of a field (default).

sourceReferenceField

String sourceReferenceField

The document field name containing the value to be stored in IDOL DREREFERENCE field. Set to null in order to use the document reference instead of a field (default).

url

String url

The IDOL/DIH/CFS index action URL. Default is http://localhost:9001.

Package com.norconex.committer.neo4j

Class com.norconex.committer.neo4j.Neo4jCommitterConfig

class Neo4jCommitterConfig extends BaseBatchCommitterConfig implements Serializable

serialVersionUID:

1L

Serialized Fields

credentials

Credentials credentials

Connection credentials.

database

String database

A database name when using one other than the default one.

deleteCypher

String deleteCypher

Cypher query for adding relationships to Neo4j. Typically, you want to use "nodeIdProperty" value to delete nodes having a matching property value. The query parameters correspond the document fields.

multiValuesJoiner

String multiValuesJoiner

One or more characters to join multi-value fields. Default is "|".

nodeContentProperty

String nodeContentProperty

Optional property name where to store the document content in Neo4j graph entries. Use it as a cypher parameter in your configured "upsertCypher" query. Default is "content".

nodeIdProperty

String nodeIdProperty

Optional property name where to store the document reference in Neo4j graph entries. Use it as a cypher parameter to uniquely identify your graph entries in your configured "upsertCypher" and "deleteCypher" queries. Default is "id".

optionalParameters

Set<String> optionalParameters

Comma-separated list of parameter names that can be missing when creating the query. They will be set to NullValue) to avoid client exception for missing parameters.

upsertCypher

String upsertCypher

Cypher query for adding relationships to Neo4j. Typically, you want to use MERGE and the "nodeIdProperty" value to update existing entries matching the ID. In order to delete all nodes related to an entry, make sure to add the ID property on all appropriate nodes. The query parameters correspond the document fields.

uri

String uri

Required connection URI. E.g., "bolt://localhost:7687".

Package com.norconex.committer.sql

Class com.norconex.committer.sql.SqlCommitterConfig

class SqlCommitterConfig extends BaseBatchCommitterConfig implements Serializable

serialVersionUID:

1L

Serialized Fields

connectionUrl

String connectionUrl

JDBC connection URL.

createFieldSQL

String createFieldSQL

The ALTER statement used to create missing table fields. The {tableName} variable will be replaced with the configuration option of the same name. The {fieldName} variable will be replaced by newly encountered field names.

createTableSQL

String createTableSQL

The CREATE statement used to create a table if it does not already exist. If you need fields of specific data types, specify them here. You can use the variable placeholders {tableName} and {primaryKey} which will be replaced with the configuration option of the same name. If you do not use those variables, make sure you use the same names.

credentials

Credentials credentials

Database credentials.

driverClass

String driverClass

Class name of the JDBC driver to use.

driverPath

String driverPath

Path to JDBC driver. Not required if already in classpath.

fixFieldNames

boolean fixFieldNames

Whether to attempt preventing insertion errors by converting characters that are not underscores or alphanumeric to underscores. Will also remove all non-alphabetic characters that prefixes a field name.

fixFieldValues

boolean fixFieldValues

Attempts to prevent insertion errors by truncating values that are larger than their defined maximum field length.

multiValuesJoiner

String multiValuesJoiner

One or more characters to join multi-value fields. Default is "|".

primaryKey

String primaryKey

The name of the table primary key field where the document reference will be stored, unless it is already set by a field of the same name in the source document. At a minimum, this "primaryKey" field name should be "unique", ideally indexed.

properties

Properties properties

Extra connection properties as per database vendor.

tableName

String tableName

The target database table name where documents will be committed.

targetContentField

String targetContentField

Table field name where to store the document content stream. Make it empty or a self-closed tag if you do not want to store the document content. Since document content can sometimes be quite large, a CLOB field is usually advised. If there is already a document field with the same name, that document field takes precedence and the content stream is ignored. Default is "content".

Package com.norconex.crawler.core

Exception Class com.norconex.crawler.core.CrawlerException

class CrawlerException extends RuntimeException implements Serializable

serialVersionUID:

1L

Package com.norconex.crawler.core.cli

Exception Class com.norconex.crawler.core.cli.CliException

class CliException extends CrawlerException implements Serializable

serialVersionUID:

1L

Package com.norconex.crawler.core.cluster

Exception Class com.norconex.crawler.core.cluster.ClusterException

class ClusterException extends CrawlerException implements Serializable

serialVersionUID:

1L

Class com.norconex.crawler.core.cluster.QueryFilter

class QueryFilter extends Object implements Serializable

serialVersionUID:

1L

Serialized Fields

fieldName

String fieldName

fieldValue

Serializable fieldValue

Package com.norconex.crawler.core.cluster.admin

Exception Class com.norconex.crawler.core.cluster.admin.ClusterAdminException

class ClusterAdminException extends CrawlerException implements Serializable

serialVersionUID:

1L

Package com.norconex.crawler.core.cluster.pipeline

Exception Class com.norconex.crawler.core.cluster.pipeline.PipelineException

class PipelineException extends CrawlerException implements Serializable

serialVersionUID:

1L

Class com.norconex.crawler.core.cluster.pipeline.StepRecord

class StepRecord extends Object implements Serializable

serialVersionUID:

1L

Serialized Fields

pipelineId

String pipelineId

runId

String runId

status

PipelineStatus status

stepId

String stepId

updatedAt

long updatedAt

Package com.norconex.crawler.core.event

Class com.norconex.crawler.core.event.CrawlerEvent

class CrawlerEvent extends Event implements Serializable

serialVersionUID:

1L

Package com.norconex.crawler.core.fetch

Exception Class com.norconex.crawler.core.fetch.FetchException

class FetchException extends IOException implements Serializable

serialVersionUID:

1L

Package com.norconex.crawler.core.ledger

Class com.norconex.crawler.core.ledger.CrawlerEntry

class CrawlerEntry extends Object implements Serializable

serialVersionUID:

1L

Serialization Methods

readObject

private void readObject(ObjectInputStream in)
                 throws IOException,
ClassNotFoundException

Throws:

IOException

ClassNotFoundException

writeObject

private void writeObject(ObjectOutputStream out)
                  throws IOException

Throws:

IOException

Serialized Fields

contentChecksum

String contentChecksum

contentType

ContentType contentType

deleted

boolean deleted

depth

int depth

Typically, the number of "hops" or directories to get to a file.

lastModified

ZonedDateTime lastModified

metaChecksum

String metaChecksum

orphan

boolean orphan

processedAt

ZonedDateTime processedAt

processingAt

ZonedDateTime processingAt

processingOutcome

ProcessingOutcome processingOutcome

processingStatus

@NonNull ProcessingStatus processingStatus

queuedAt

ZonedDateTime queuedAt

reference

String reference

referenceTrail

List<String> referenceTrail

Original references before normalization, redirect, etc.

Class com.norconex.crawler.core.ledger.ProcessingOutcome

class ProcessingOutcome extends Object implements Serializable

serialVersionUID:

6542269270632505768L

Serialized Fields

outcome

String outcome

Package com.norconex.crawler.fs.doc

Class com.norconex.crawler.fs.doc.FsCrawlerDocContext

class FsCrawlerDocContext extends FsCrawlerEntry implements Serializable

serialVersionUID:

1L

Package com.norconex.crawler.fs.ledger

Class com.norconex.crawler.fs.ledger.FsCrawlerEntry

class FsCrawlerEntry extends CrawlerEntry implements Serializable

serialVersionUID:

1L

Serialized Fields

file

boolean file

folder

boolean folder

Package com.norconex.crawler.web.doc

Class com.norconex.crawler.web.doc.WebProcessingOutcome

class WebProcessingOutcome extends ProcessingOutcome implements Serializable

serialVersionUID:

1L

Package com.norconex.crawler.web.doc.operations.sitemap

Class com.norconex.crawler.web.doc.operations.sitemap.SitemapRecord

class SitemapRecord extends Object implements Serializable

serialVersionUID:

1L

Serialized Fields

crawlDate

ZonedDateTime crawlDate

lastModified

ZonedDateTime lastModified

location

String location

Package com.norconex.crawler.web.fetch.impl.webdriver

Class com.norconex.crawler.web.fetch.impl.webdriver.WebDriverBrowser.CustomDriverOptions

class CustomDriverOptions extends org.openqa.selenium.remote.AbstractDriverOptions<WebDriverBrowser.CustomDriverOptions> implements Serializable

serialVersionUID:

1L

Package com.norconex.crawler.web.ledger

Class com.norconex.crawler.web.ledger.WebCrawlerEntry

class WebCrawlerEntry extends CrawlerEntry implements Serializable

serialVersionUID:

1L

Serialized Fields

etag

String etag

The HTTP ETag.

Since:

3.0.0

fromSitemap

boolean fromSitemap

Whether this record was obtained from parsing a sitemap. Set by the queue pipeline so implementors should not have to set it themselves.

httpReasonPhrase

String httpReasonPhrase

HTTP reason phrase from the last fetch response.

httpStatusCode

int httpStatusCode

HTTP status code from the last fetch response.

redirectTarget

String redirectTarget

redirectTrail

List<String> redirectTrail

referencedUrls

List<String> referencedUrls

referrerLinkMetadata

String referrerLinkMetadata

referrerReference

String referrerReference

sitemapChangeFreq

String sitemapChangeFreq

The document change frequency according to sitemap.

sitemapLastMod

ZonedDateTime sitemapLastMod

The document last modified date according to sitemap.

sitemapPriority

Float sitemapPriority

The document priority according to sitemap.

urlRoot

String urlRoot

Package com.norconex.importer

Class com.norconex.importer.ImporterEvent

class ImporterEvent extends Event implements Serializable

serialVersionUID:

1L

Serialized Fields

parseState

ParseState parseState

Gets the document parse state (never null).

Exception Class com.norconex.importer.ImporterException

class ImporterException extends IOException implements Serializable

serialVersionUID:

1L

Exception Class com.norconex.importer.ImporterRuntimeException

class ImporterRuntimeException extends RuntimeException implements Serializable

serialVersionUID:

1L

Package com.norconex.importer.handler

Exception Class com.norconex.importer.handler.DocHandlerException

class DocHandlerException extends ImporterException implements Serializable

serialVersionUID:

1L

Package com.norconex.importer.handler.parser.impl

Class com.norconex.importer.handler.parser.impl.EmbeddedConfig

class EmbeddedConfig extends Object implements Serializable

serialVersionUID:

1L

Serialized Fields

maxEmbeddedDepth

int maxEmbeddedDepth

Maximum detph of embeded documents to extract. Zero is the root document (original container).

skipEmbeddedContentTypes

List<TextMatcher> skipEmbeddedContentTypes

skipEmbeddedOfContentTypes

List<TextMatcher> skipEmbeddedOfContentTypes

splitContentTypes

List<TextMatcher> splitContentTypes

Class com.norconex.importer.handler.parser.impl.XfdlTikaParser

class XfdlTikaParser extends org.apache.tika.parser.xml.XMLParser implements Serializable

serialVersionUID:

1L