Model family notes

Most models need nothing beyond mistralrs run -m <id> (see Run models). This page collects the per-family exceptions. The full architecture inventory is in the supported models reference.

Reasoning controls

reasoning_effort accepts off, low, medium, high, and xhigh; none is an alias for off. If no effort is set, mistral.rs leaves it unspecified and enables thinking. An explicit positive effort enables thinking, while off disables it. The legacy enable_thinking boolean remains supported, but contradictory explicit values return an error.

These controls are template inputs, not sampling parameters. mistral.rs passes an explicit effort under both reasoning_effort and the template compatibility name reasoning_strength. Each model’s template decides whether it distinguishes every positive tier.

Thinking models (Qwen3, SmolLM3)

Qwen3 and SmolLM3 are hybrid reasoning models; their chat templates enable thinking by default. Toggle it per request. Inline /think and /no_think prompt tags work everywhere; the --thinking flag and the enable_thinking field do the same without editing user text (true forces on and false forces off).

CLI

mistralrs run --thinking false -m Qwen/Qwen3-4B

--thinking and --reasoning-effort apply to both one-shot and interactive use. Prompt tags also work inline:

How many rs are in blueberry? /no_think
Are you sure? /think

HTTP

curl http://localhost:1234/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "default",
    "messages": [{"role": "user", "content": "How many rs are in blueberry?"}],
    "enable_thinking": false
  }'

Python

runner.send_chat_completion_request(
    ChatCompletionRequest(
        model="default",
        messages=[{"role": "user", "content": "How many rs are in blueberry?"}],
        enable_thinking=False,
    )
)

Qwen3 also publishes FP8 pre-quantized checkpoints; pass the FP8 model ID directly when you want those weights instead of runtime ISQ (in-situ quantization).

MoE models and MoQE

MoE (Mixture of Experts) families (DeepSeek V2/V3, GLM-4.7, GLM-4.7-Flash, Phi 3.5 MoE, Qwen3 MoE, Qwen3-VL MoE, Qwen3.5/Qwen3.6 MoE, LFM2/LFM2.5 MoE) support MoQE (Mixture of Quantized Experts): quantizing only the routed experts, which dominate memory, while leaving the rest of the model alone. Enable it when applying ISQ with --isq-organization moqe:

mistralrs run --isq 4 --isq-organization moqe -m Qwen/Qwen3-30B-A3B

MoQE also applies when generating UQFF with mistralrs quantize. It does not rewrite an already-quantized GGUF or UQFF selected by --quant; those weights load as published.

In the Python SDK, pass organization=IsqOrganization.MoQE inside Which.Plain(...) or Which.MultimodalPlain(...). Expect small output differences between quantization levels: router decisions are sensitive to numerical noise.

MLA models (DeepSeek V2/V3, GLM-4.7-Flash)

DeepSeek V2, DeepSeek V3 (including non-distill R1, which uses the V3 architecture), and GLM-4.7-Flash use MLA (Multi-head Latent Attention). The KV cache stores a low-dimensional latent instead of full K/V, so the cache footprint is substantially smaller than standard attention at the same context length.

On CUDA (Unix builds), a specialized MLA decode kernel is used when all of the following hold:

• single-token decode (no attention mask, sequence length 1);
• paged attention enabled;
• FlashInfer (NVIDIA’s attention-kernel library) paged metadata available.

A parallel fast path covers prefill with prefix caching (paged attention on, CUDA device). Otherwise the generic attention path reconstructs the latent per step.

MISTRALRS_NO_MLA=1 forces the generic path; use it when debugging suspected MLA kernel issues, and try --paged-attn off as a sanity check for unexpected paged-attention behavior. Background: the DeepSeek V2 paper.

GPT-OSS

GPT-OSS experts are stored pre-quantized in MXFP4 (4-bit microscaling float), and its attention uses per-head sinks. Load it without a quantization flag first:

mistralrs run -m openai/gpt-oss-20b

ISQ applies only to the attention layers (and lm_head); the expert weights are already quantized.

Qwen3 Next

Qwen3 Next mixes Gated Delta Network (linear attention) layers with full softmax attention, so its cost profile at long contexts differs from a pure softmax model. Qwen3-Coder-Next checkpoints use the same loader.

Qwen3.5 and Qwen3.6

Qwen3.5 and Qwen3.6 dense and MoE checkpoints are supported. Qwen3.6 shares the Qwen3.5 HF model_type, so it loads through the same Qwen3_5 and Qwen3_5Moe paths.

Granite 4.0 (GraniteMoeHybrid)

IBM Granite 4.0 checkpoints (e.g. ibm-granite/granite-4.0-micro) mix Mamba-2 recurrent layers with attention layers. They load through auto-detection like any other text model.

LFM2 and LFM2.5

LiquidAI LFM2 and LFM2.5 dense and MoE checkpoints are supported. LFM2-VL and LFM2.5-VL are supported for image input.

The official repos ship chat_template.jinja, including the Liquid tool-call format. It is auto-detected like other model-native tool-call formats.

Gemma

Gemma repos are gated: accept the license on the Hugging Face model page, then authenticate with mistralrs login.

Gemma 4 accepts image, audio, and video parts mixed in one message, and enforces its tool-call format through constrained decoding by default; see tool calling.

Muse Glimmer

Muse Glimmer loads through normal model auto-detection:

mistralrs run -m meta-models/Muse-Glimmer-30B

The native safetensors checkpoint accepts text, images, and video; it does not accept audio. Video is represented as timestamped frame groups, and the model card notes that the model was trained primarily for image rather than video understanding.

The model’s ATEM chat template is detected automatically. Tool calling supports automatic, required, named, and parallel calls through the same HTTP, Python, Rust, and agent APIs as other tool-capable models. Muse reads the reasoning_strength template variable, so mistral.rs maps the public reasoning_effort control to it. The published Muse tiers are low, medium, high, and xhigh; omission uses the template’s high default.

ISQ, UQFF, paged attention, prefix caching, tensor parallelism, and text-backbone LoRA use the standard multimodal model paths. For GGUF-specific requirements and its image-only projector limitation, see GGUF compatibility.

MatFormer

MatFormer-trained models encode multiple model sizes in one checkpoint; the desired slice is selected at load time with two values:

• matformer_config_path: path to the slice config file (CSV or JSON) shipped with the model card.
• matformer_slice_name: the named slice within that file.

Without these, the default (full) configuration loads. Gemma 3n (google/gemma-3n-E4B-it) is the MatFormer model in the supported list; the bundled matformer_configs/gemma3n.csv contains the full E4B configuration, the official E2B slice, and intermediate E1.96B-E3.79B slices:

mistralrs run -m google/gemma-3n-E4B-it \
  --matformer-config-path matformer_configs/gemma3n.csv \
  --matformer-slice-name "Config for E2.49B (block-level)"

The same slice selection is available on every surface:

• CLI: --matformer-config-path / --matformer-slice-name on run, serve, and bench.
• TOML configs: matformer_config_path / matformer_slice_name.
• Python: matformer_config_path / matformer_slice_name on the Which selectors.
• Rust SDK: with_matformer_config_path / with_matformer_slice_name on the model builders.

Use the full configuration for quality and smaller slices for constrained devices.

Mistral Small tool calling

Mistral Small 3 checkpoints can do tool calling, but some repos do not ship the right chat template. Use the bundled one:

mistralrs serve --quant 4 \
  --jinja-explicit chat_templates/mistral_small_tool_call.jinja \
  -m mistralai/Mistral-Small-3.2-24B-Instruct-2506

LLaVA chat templates

Mistral-backed LLaVA checkpoints work with the default template. Vicuna-backed checkpoints need the Vicuna template:

mistralrs run -m llava-hf/llava-v1.6-vicuna-7b-hf \
  -c chat_templates/vicuna.json --image photo.jpg -i "Describe this image"

Qwen3-VL

Per-request video frame-sampling overrides are not exposed. In multi-turn conversations reusing prefix cache entries, pixel inputs are narrowed per turn by grid count, not image count.

Llama 4

Llama 4 Scout supports up to 10M tokens of context. Using the full window requires paged attention with a large memory budget, generally with multi-GPU tensor parallelism.

Multimodal device mapping

For most multimodal models the text backbone holds most of the parameters, so device mapping and topology apply mainly to the text portion; the vision, audio, or video encoder stays on its supported device path.

Phi

Phi 3.5 Vision works best with a single image; multiple images are resized together. Phi 4 Multimodal accepts audio and image parts in the same message. Phi 3.5 MoE has 16 experts and routes each token to 2 of them; it benefits from MoQE.

DiffusionGemma

Block-diffusion generation has its own page: block-diffusion models.

Reporting issues

File an issue on GitHub, with a reproducer when possible.