Class TextEmbeddingTransformerConfig

java.lang.Object
com.norconex.importer.handler.BaseLabelableConfig
com.norconex.importer.handler.BaseDocHandlerConfig
com.norconex.importer.handler.transformer.impl.TextEmbeddingTransformerConfig
All Implemented Interfaces:
Labelable

public class TextEmbeddingTransformerConfig extends BaseDocHandlerConfig

Calls an embeddings API and stores the resulting vector as a document field. Targets the "OpenAI-compatible" /embeddings request and response shape, which is also spoken by Ollama, self-hosted embedding servers such as vLLM, and proxies such as LiteLLM that front many providers behind that same shape — so this one handler works with all of them without provider-specific code, by pointing getApiUrl() at whichever you run. The crawler calls the service you choose; it does not run an embedding model itself.

Embeds the document's body (its text content at the point this handler runs), so it should run post-parse — after TextChunkSplitter if documents are being chunked first, which is recommended for anything longer than a paragraph or two.

The vector is stored as a multi-valued field (one value per dimension). For Elasticsearch and OpenSearch, set the committer's jsonFieldsPattern to match the target field so the values are sent as JSON numbers instead of quoted strings. Solr's DenseVectorField accepts the values as they are sent.

The crawler compares a document's content checksum with the previous crawl only after the importer has run, so a document it cannot recognize as unchanged sooner (by default on the web crawler, through the Last-Modified header, before download) still reaches this handler before being found unchanged and left out of the commit. It is not necessarily embedded again, though: this handler caches each result by a hash of the exact text together with the model and endpoint that produced it, so unchanged text is not re-sent to the API — whether it recurs on a later crawl or as boilerplate shared by many documents in this same crawl. A crawler backs that cache with its own persistent, cluster-aware store, at no extra setup; running the importer standalone (e.g., from the command line), every call reaches the API.

See Also:
  • Field Details

    • DEFAULT_TARGET_FIELD

      public static final String DEFAULT_TARGET_FIELD
      See Also:
    • DEFAULT_TIMEOUT_SECONDS

      public static final int DEFAULT_TIMEOUT_SECONDS
      How long to wait for the embeddings API to respond, in seconds. A value below 1 means the default, 60.
      See Also:
  • Constructor Details

    • TextEmbeddingTransformerConfig

      public TextEmbeddingTransformerConfig()
  • Method Details

    • getApiUrl

      public String getApiUrl()
      The embeddings endpoint URL, e.g. https://api.openai.com/v1/embeddings or, for a local Ollama server, http://localhost:11434/v1/embeddings.
    • getApiKey

      public String getApiKey()
      The API key, sent as an Authorization: Bearer header. Leave blank for servers that do not require one (e.g., a local Ollama instance).
    • getModel

      public String getModel()
      The embedding model name to request, e.g. text-embedding-3-small.
    • getTargetField

      public String getTargetField()
      The metadata field the resulting vector is stored into. Blank means the default, "embedding".
    • getTimeoutSeconds

      public int getTimeoutSeconds()
    • setApiUrl

      public TextEmbeddingTransformerConfig setApiUrl(String apiUrl)
      The embeddings endpoint URL, e.g. https://api.openai.com/v1/embeddings or, for a local Ollama server, http://localhost:11434/v1/embeddings.
      Returns:
      this.
    • setApiKey

      public TextEmbeddingTransformerConfig setApiKey(String apiKey)
      The API key, sent as an Authorization: Bearer header. Leave blank for servers that do not require one (e.g., a local Ollama instance).
      Returns:
      this.
    • setModel

      public TextEmbeddingTransformerConfig setModel(String model)
      The embedding model name to request, e.g. text-embedding-3-small.
      Returns:
      this.
    • setTargetField

      public TextEmbeddingTransformerConfig setTargetField(String targetField)
      The metadata field the resulting vector is stored into. Blank means the default, "embedding".
      Returns:
      this.
    • setTimeoutSeconds

      public TextEmbeddingTransformerConfig setTimeoutSeconds(int timeoutSeconds)
      Returns:
      this.
    • equals

      public boolean equals(Object o)
      Overrides:
      equals in class BaseLabelableConfig
    • canEqual

      protected boolean canEqual(Object other)
      Overrides:
      canEqual in class BaseLabelableConfig
    • hashCode

      public int hashCode()
      Overrides:
      hashCode in class BaseLabelableConfig
    • toString

      public String toString()
      Overrides:
      toString in class BaseLabelableConfig