Use embedding models

Embedding models map text to dense vectors for semantic search, reranking, clustering, and downstream retrieval. mistral.rs serves embeddings through the standard OpenAI POST /v1/embeddings endpoint, so any tool that already targets that endpoint (LangChain, LlamaIndex, vector stores) works unchanged.

Loading an embedding model

Two regularly tested options:

• google/embeddinggemma-300m: 300M parameters, 768-dim vectors. Good general-purpose default.
• Qwen/Qwen3-Embedding-0.6B: 600M parameters, 1024-dim vectors.

mistralrs serve -m google/embeddinggemma-300m

Use Qwen/Qwen3-Embedding-0.6B the same way:

mistralrs serve -m Qwen/Qwen3-Embedding-0.6B

Requesting an embedding

HTTP

curl http://localhost:1234/v1/embeddings \
  -H "Content-Type: application/json" \
  -d '{
    "model": "default",
    "input": "The cat sat on the mat."
  }'

The response includes the vector in embedding:

{
  "object": "list",
  "data": [{
    "object": "embedding",
    "index": 0,
    "embedding": [0.123, -0.456, 0.789, ...]
  }],
  "model": "default",
  "usage": {"prompt_tokens": 7, "total_tokens": 7}
}

Python

from mistralrs import EmbeddingArchitecture, EmbeddingRequest, Runner, Which

runner = Runner(
    which=Which.Embedding(
        model_id="google/embeddinggemma-300m",
        arch=EmbeddingArchitecture.EmbeddingGemma,
    )
)

embeddings = runner.send_embedding_request(
    EmbeddingRequest(
        input=[
            "task: search result | query: What is graphene?",
            "task: search result | query: What is an apple?",
        ],
        truncate_sequence=True,
    )
)
print(len(embeddings), len(embeddings[0]))

For Qwen3-Embedding, set arch=EmbeddingArchitecture.Qwen3Embedding and drop the EmbeddingGemma prompt prefixes.

Rust

use mistralrs::{EmbeddingModelBuilder, EmbeddingRequest};

let model = EmbeddingModelBuilder::new("google/embeddinggemma-300m")
    .build()
    .await?;

let embeddings = model
    .generate_embeddings(
        EmbeddingRequest::builder()
            .add_prompt("task: search result | query: What is graphene?"),
    )
    .await?;
println!("{:?}", embeddings.first());

Batching

Pass a list of strings to embed many at once:

curl http://localhost:1234/v1/embeddings \
  -H "Content-Type: application/json" \
  -d '{
    "model": "default",
    "input": [
      "The cat sat on the mat.",
      "A dog chased a squirrel.",
      "A raven croaked from the fencepost."
    ]
  }'

The data array has one entry per input in input order.

Request fields

Beyond model and input:

• encoding_format: "float" (default) returns float arrays. "base64" returns each vector as base64-encoded little-endian f32 bytes. This matches OpenAI's compact form.
• dimensions: not supported. Passing it returns a validation error rather than silently truncating.
• truncate_sequence: mistral.rs extension. true truncates inputs that exceed the model's context length instead of erroring.

Normalization

Whether vectors are L2-normalized is model-dependent: mistral.rs applies the modules the model ships in modules.json. EmbeddingGemma includes a Normalize module, so its output is already unit-norm; Qwen3-Embedding does not, so its vectors are returned unnormalized. If you compute cosine similarity as a raw dot product and your model does not normalize, normalize on the client first.

To normalize in Python:

import numpy as np

v = np.array(response["data"][0]["embedding"])
v_normalized = v / np.linalg.norm(v)

Many vector stores (FAISS, pgvector) handle normalization internally.

EmbeddingGemma prompts

EmbeddingGemma works best when the input is prefixed for the task:

| Use case | Prompt form | |---|---| | Retrieval query | task: search result \| query: <query> | | Retrieval document | title: <title or none> \| text: <document> | | Question answering | task: question answering \| query: <question> | | Fact verification | task: fact checking \| query: <claim> | | Classification | task: classification \| query: <text> | | Clustering | task: clustering \| query: <text> | | Semantic similarity | task: sentence similarity \| query: <text> | | Code retrieval | task: code retrieval \| query: <query> |

Example:

curl http://localhost:1234/v1/embeddings \
  -H "Content-Type: application/json" \
  -d '{
    "model": "default",
    "input": [
      "task: search result | query: What is graphene?",
      "title: none | text: Graphene is a single layer of carbon atoms."
    ]
  }'

Qwen3-Embedding does not require these prefixes, but task-specific prefixes can still help keep a retrieval system consistent.

Using the vectors

Standard pipeline:

1. Embed a corpus offline and store vectors in a vector database (FAISS, Qdrant, pgvector, Pinecone).
2. At query time, embed the query with the same model.
3. Search the store for nearest neighbors.
4. Optionally rerank top results with a reranker.
5. Feed retrieved documents as language model context.

mistral.rs handles steps 1, 2, and (with a reranker) 4. The rest is the vector store and application logic. The web search guide covers using embeddings to rerank search results within an agent.