Package com.norconex.importer.handler.transformer.impl

Class LanguageTransformerConfig

java.lang.Object
com.norconex.importer.handler.BaseLabelableConfig
com.norconex.importer.handler.BaseDocHandlerConfig
com.norconex.importer.handler.transformer.impl.LanguageTransformerConfig
All Implemented Interfaces:
Labelable, ChunkedTextSupport
public class LanguageTransformerConfig extends BaseDocHandlerConfig implements ChunkedTextSupport

Detects a document language based on Apache Tika language detection capability. It adds the detected language to the "document.language" metadata field. Optionally adds all potential languages detected with their probability score as well as additional fields following this pattern:

 document.language.<rank>.tag
 document.language.<rank>.probability

<rank> is to indicate the match order, based on match probability score (starting at 1).

This tagger can be used both as a pre-parse (on text only) or post-parse handler.

Accuracy:

To obtain optimal detection, long enough text is expected. The default detection algorithm is optimized for document with lots of text. This tagger relies on Tika language detection capabilities and future versions may provide better precision for documents made of short text (e.g. tweets, comments, etc).

If you know what mix of languages are used by your site(s), you can increase accuracy in many cases by limiting the set of languages supported for detection.

Supported Languages:

Languages are represented as code values. As of 2.6.0, at least the following 70 languages are supported by the Tika version used:

  • af Afrikaans
  • an Aragonese
  • ar Arabic
  • ast Asturian
  • be Belarusian
  • br Breton
  • ca Catalan
  • bg Bulgarian
  • bn Bengali
  • cs Czech
  • cy Welsh
  • da Danish
  • de German
  • el Greek
  • en English
  • es Spanish
  • et Estonian
  • eu Basque
  • fa Persian
  • fi Finnish
  • fr French
  • ga Irish
  • gl Galician
  • gu Gujarati
  • he Hebrew
  • hi Hindi
  • hr Croatian
  • ht Haitian
  • hu Hungarian
  • id Indonesian
  • is Icelandic
  • it Italian
  • ja Japanese
  • km Khmer
  • kn Kannada
  • ko Korean
  • lt Lithuanian
  • lv Latvian
  • mk Macedonian
  • ml Malayalam
  • mr Marathi
  • ms Malay
  • mt Maltese
  • ne Nepali
  • nl Dutch
  • no Norwegian
  • oc Occitan
  • pa Punjabi
  • pl Polish
  • pt Portuguese
  • ro Romanian
  • ru Russian
  • sk Slovak
  • sl Slovene
  • so Somali
  • sq Albanian
  • sr Serbian
  • sv Swedish
  • sw Swahili
  • ta Tamil
  • te Telugu
  • th Thai
  • tl Tagalog
  • tr Turkish
  • uk Ukrainian
  • ur Urdu
  • vi Vietnamese
  • yi Yiddish
  • zh-cn Simplified Chinese
  • zh-tw Traditional Chinese

It is possible more will be supported automatically with future Tika upgrades.

If you do not restrict the list of language candidates to detect, the default behavior is to try match all languages currently supported.

See Also:

  • Constructor Details

    • LanguageTransformerConfig

      public LanguageTransformerConfig()

  • Method Details

    • getLanguages

      public List<String> getLanguages()
      The language candidates for language detection.
      Returns:
      languages to consider for detection

    • setLanguages

      public LanguageTransformerConfig setLanguages(List<String> languages)
      The language candidates for language detection.
      Parameters:
      languages - languages to consider for detection

    • getFieldMatcher

      public TextMatcher getFieldMatcher()
      Gets a matcher for fields to use to detect language. When not specified (default), use the document content instead.
      Specified by:
      getFieldMatcher in interface ChunkedTextSupport
      Returns:
      field matcher

    • setFieldMatcher

      public LanguageTransformerConfig setFieldMatcher(TextMatcher fieldMatcher)
      Sets a matcher for fields to use to detect language. When not specified (default), use the document content instead.
      Parameters:
      fieldMatcher - field matcher

    • getMaxReadSize

      public int getMaxReadSize()
      Specified by:
      getMaxReadSize in interface ChunkedTextSupport

    • getSourceCharset

      public Charset getSourceCharset()
      Specified by:
      getSourceCharset in interface ChunkedTextSupport

    • isKeepProbabilities

      public boolean isKeepProbabilities()
      Whether to keep the match probabilities for each languages detected. Default is false.
      Returns:
      true if probability kept

    • getFallbackLanguage

      public String getFallbackLanguage()
      The fallback language when none are detected. Default behavior is to not tag incoming documents with a language field when no detection occurs.
      Returns:
      the fallback language

    • setMaxReadSize

      public LanguageTransformerConfig setMaxReadSize(int maxReadSize)
      Returns:
      this.

    • setSourceCharset

      public LanguageTransformerConfig setSourceCharset(Charset sourceCharset)
      Returns:
      this.

    • setKeepProbabilities

      public LanguageTransformerConfig setKeepProbabilities(boolean keepProbabilities)
      Whether to keep the match probabilities for each languages detected. Default is false.
      Parameters:
      keepProbabilities - true to keep probabilities
      Returns:
      this.

    • setFallbackLanguage

      public LanguageTransformerConfig setFallbackLanguage(String fallbackLanguage)
      The fallback language when none are detected. Default behavior is to not tag incoming documents with a language field when no detection occurs.
      Parameters:
      fallbackLanguage - the default languages when no detection
      Returns:
      this.

    • equals

      public boolean equals(Object o)
      Overrides:
      equals in class BaseLabelableConfig

    • canEqual

      protected boolean canEqual(Object other)
      Overrides:
      canEqual in class BaseLabelableConfig

    • hashCode

      public int hashCode()
      Overrides:
      hashCode in class BaseLabelableConfig

    • toString

      public String toString()
      Overrides:
      toString in class BaseLabelableConfig