TranslatorSplitter

Translate documents using one of the supported translation API.

importerJavadoc: TranslatorSplitter

Notes

Translate documents using one of the supported translation APIs. The following lists the supported APIs and their required authentication properties:

• microsoft

  • clientId
  • clientSecret

• google

  • apiKey

• lingo24

  • userKey

• moses

  • smtPath
  • scriptPath

• yandex

  • apiKey

For example, the Microsoft Translation API requires a client ID and a client secret, both obtained from the Microsoft Azure Marketplace using your Microsoft account.

Translated documents will have the original language stored in the field:
document.translatedFrom

Note: This class is not a document "splitter" in the strict sense, but like splitters, translation creates child documents for each translated version. The original document remains the parent, and each translation becomes a child document.

Examples

Translates each document into English and French with the Google API, taking the original language from a langField field. Two child documents are created per document.

YAML

handler:
  class: TranslatorSplitter
  api: google
  sourceLanguageField: langField
  targetLanguages:
    - en
    - fr
  apiKey: MYKEYHERE

JSON

{
  "handler": {
    "class": "TranslatorSplitter",
    "api": "google",
    "sourceLanguageField": "langField",
    "targetLanguages": [
      "en",
      "fr"
    ],
    "apiKey": "MYKEYHERE"
  }
}

XML

<handler>
  <class>TranslatorSplitter</class>
  <api>google</api>
  <sourceLanguageField>langField</sourceLanguageField>
  <targetLanguages>
    <targetLanguage>en</targetLanguage>
    <targetLanguage>fr</targetLanguage>
  </targetLanguages>
  <apiKey>MYKEYHERE</apiKey>
</handler>

Usage

Full configuration skeleton, with every option and its default

YAML

class: TranslatorSplitter
api: string
apiKey: string
clientId: string
clientSecret: string
discardOriginal: false
fieldMatcher:
  ignoreCase: false
  ignoreDiacritic: false
  matchEmpty: false
  method: BASIC
  negateMatches: false
  partial: false
  pattern: string
  replaceAll: false
  trim: false
ignoreNonTranslatedFields: false
label: string
scriptPath: string
smtPath: string
sourceLanguage: string
sourceLanguageField: string
targetLanguages:
  - string
userKey: string

JSON

{
  "class": "TranslatorSplitter",
  "api": "string",
  "apiKey": "string",
  "clientId": "string",
  "clientSecret": "string",
  "discardOriginal": false,
  "fieldMatcher": {
    "ignoreCase": false,
    "ignoreDiacritic": false,
    "matchEmpty": false,
    "method": "BASIC",
    "negateMatches": false,
    "partial": false,
    "pattern": "string",
    "replaceAll": false,
    "trim": false
  },
  "ignoreNonTranslatedFields": false,
  "label": "string",
  "scriptPath": "string",
  "smtPath": "string",
  "sourceLanguage": "string",
  "sourceLanguageField": "string",
  "targetLanguages": [
    "string"
  ],
  "userKey": "string"
}

XML

<handler>
  <class>TranslatorSplitter</class>
  <api>string</api>
  <apiKey>string</apiKey>
  <clientId>string</clientId>
  <clientSecret>string</clientSecret>
  <discardOriginal>false</discardOriginal>
  <fieldMatcher>
    <ignoreCase>false</ignoreCase>
    <ignoreDiacritic>false</ignoreDiacritic>
    <matchEmpty>false</matchEmpty>
    <method>BASIC</method>
    <negateMatches>false</negateMatches>
    <partial>false</partial>
    <pattern>string</pattern>
    <replaceAll>false</replaceAll>
    <trim>false</trim>
  </fieldMatcher>
  <ignoreNonTranslatedFields>false</ignoreNonTranslatedFields>
  <label>string</label>
  <scriptPath>string</scriptPath>
  <smtPath>string</smtPath>
  <sourceLanguage>string</sourceLanguage>
  <sourceLanguageField>string</sourceLanguageField>
  <targetLanguages>
    <targetLanguage>string</targetLanguage>
  </targetLanguages>
  <userKey>string</userKey>
</handler>

Properties

Property	Type	Required	Default
api	string	No	-
apiKey	string	No	-
clientId	string	No	-
clientSecret	string	No	-
discardOriginal	boolean	No	false
fieldMatcher	TextMatcher	No	TextMatcher
ignoreNonTranslatedFields	boolean	No	false
label	string	No	-
scriptPath	string	No	-
smtPath	string	No	-
sourceLanguage	string	No	-
sourceLanguageField	string	No	-
targetLanguages	string[]	No	-
userKey	string	No	-

Property Details

api

Which translation service to use: microsoft, google, lingo24, moses or yandex. Each expects its own credentials, listed above — the other authentication properties are ignored.

apiKey

API key for the translation service. Used by google and yandex.

clientId

Client ID for the microsoft API, obtained on Microsoft Azure Marketplace with your Microsoft account.

clientSecret

Client secret paired with clientId for the microsoft API.

discardOriginal

When enabled, the original document is discarded after splitting, leaving only the split sub-documents.

fieldMatcher

Matches document fields based on specified patterns and methods. Use this to restrict the handler to operate only on fields whose names satisfy the matcher criteria.

ignoreNonTranslatedFields

Whether child documents should carry only the fields that were actually translated. Default is false, which copies every field of the parent onto each child. Set it to true to keep the children lean.

label

An optional, user-supplied label for this step in the importer pipeline. It has no effect on processing — it exists purely to help identify this step, for example in large configurations or in the visual configurator.

scriptPath

Path to the moses scripts directory. Required with smtPath when api is moses, which runs locally rather than as a hosted service.

smtPath

Path to the moses statistical machine translation installation. Required with scriptPath when api is moses.

sourceLanguage

The language documents are translated from, when it is the same for the whole crawl. Use sourceLanguageField instead when it varies per document.

sourceLanguageField

Name of a document field holding the original language. Takes precedence over sourceLanguage and is the usual choice after a LanguageTransformer has detected the language.

targetLanguages

The languages to translate to. One child document is created per language, per document.

userKey

User key for the lingo24 API.