Package com.norconex.importer.handler.transformer.impl

Class TextEmbeddingTransformerConfig

java.lang.Object
com.norconex.importer.handler.BaseLabelableConfig
com.norconex.importer.handler.BaseDocHandlerConfig
com.norconex.importer.handler.transformer.impl.TextEmbeddingTransformerConfig
All Implemented Interfaces:
Labelable
public class TextEmbeddingTransformerConfig extends BaseDocHandlerConfig

Calls an embeddings API and stores the resulting vector as a document field. Targets the "OpenAI-compatible" /embeddings request and response shape, which is also spoken by Ollama, self-hosted embedding servers such as vLLM, and proxies such as LiteLLM that front many providers behind that same shape — so this one handler works with all of them without provider-specific code, by pointing getApiUrl() at whichever you run. The crawler calls the service you choose; it does not run an embedding model itself.

Embeds the document's body (its text content at the point this handler runs), so it should run post-parse — after TextChunkSplitter if documents are being chunked first, which is recommended for anything longer than a paragraph or two.

The vector is stored as a multi-valued field (one value per dimension). For Elasticsearch and OpenSearch, set the committer's jsonFieldsPattern to match the target field so the values are sent as JSON numbers instead of quoted strings. Solr's DenseVectorField accepts the values as they are sent.

The crawler compares a document's content checksum with the previous crawl only after the importer has run, so a document it cannot recognize as unchanged sooner (by default on the web crawler, through the Last-Modified header, before download) is embedded again before being found unchanged and left out of the commit.

See Also:

  • Field Details

    • DEFAULT_TARGET_FIELD

      public static final String DEFAULT_TARGET_FIELD
      See Also:

    • DEFAULT_TIMEOUT_SECONDS

      public static final int DEFAULT_TIMEOUT_SECONDS
      How long to wait for the embeddings API to respond, in seconds. A value below 1 means the default, 60.
      See Also:

  • Constructor Details

    • TextEmbeddingTransformerConfig

      public TextEmbeddingTransformerConfig()

  • Method Details

    • getApiUrl

      public String getApiUrl()
      The embeddings endpoint URL, e.g. https://api.openai.com/v1/embeddings or, for a local Ollama server, http://localhost:11434/v1/embeddings.

    • getApiKey

      public String getApiKey()
      The API key, sent as an Authorization: Bearer header. Leave blank for servers that do not require one (e.g., a local Ollama instance).

    • getModel

      public String getModel()
      The embedding model name to request, e.g. text-embedding-3-small.

    • getTargetField

      public String getTargetField()
      The metadata field the resulting vector is stored into. Blank means the default, "embedding".

    • getTimeoutSeconds

      public int getTimeoutSeconds()

    • setApiUrl

      public TextEmbeddingTransformerConfig setApiUrl(String apiUrl)
      The embeddings endpoint URL, e.g. https://api.openai.com/v1/embeddings or, for a local Ollama server, http://localhost:11434/v1/embeddings.
      Returns:
      this.

    • setApiKey

      public TextEmbeddingTransformerConfig setApiKey(String apiKey)
      The API key, sent as an Authorization: Bearer header. Leave blank for servers that do not require one (e.g., a local Ollama instance).
      Returns:
      this.

    • setModel

      public TextEmbeddingTransformerConfig setModel(String model)
      The embedding model name to request, e.g. text-embedding-3-small.
      Returns:
      this.

    • setTargetField

      public TextEmbeddingTransformerConfig setTargetField(String targetField)
      The metadata field the resulting vector is stored into. Blank means the default, "embedding".
      Returns:
      this.

    • setTimeoutSeconds

      public TextEmbeddingTransformerConfig setTimeoutSeconds(int timeoutSeconds)
      Returns:
      this.

    • equals

      public boolean equals(Object o)
      Overrides:
      equals in class BaseLabelableConfig

    • canEqual

      protected boolean canEqual(Object other)
      Overrides:
      canEqual in class BaseLabelableConfig

    • hashCode

      public int hashCode()
      Overrides:
      hashCode in class BaseLabelableConfig

    • toString

      public String toString()
      Overrides:
      toString in class BaseLabelableConfig