Run models

mistral.rs detects text, multimodal, embedding, speech, and diffusion pipelines from supported checkpoints. The same run and serve commands accept Hugging Face model IDs and local model directories.

CLI

mistralrs run -m Qwen/Qwen3-4B
mistralrs serve -m Qwen/Qwen3-4B

run opens an interactive chat (or one-shot with -i); serve starts the OpenAI-compatible server on port 1234.

Python

from mistralrs import Runner, Which, ChatCompletionRequest

runner = Runner(which=Which.Plain(model_id="Qwen/Qwen3-4B"))

res = runner.send_chat_completion_request(
    ChatCompletionRequest(
        model="default",
        messages=[{"role": "user", "content": "Hello!"}],
        max_tokens=256,
    )
)
print(res.choices[0].message.content)

arch is optional; it is detected from the model config. Full example.

Rust

use anyhow::Result;
use mistralrs::{ModelBuilder, TextMessageRole, TextMessages};

#[tokio::main]
async fn main() -> Result<()> {
    let model = ModelBuilder::new("Qwen/Qwen3-4B").with_logging().build().await?;

    let messages = TextMessages::new().add_message(TextMessageRole::User, "Hello!");
    let response = model.send_chat_request(messages).await?;
    println!("{}", response.choices[0].message.content.as_ref().unwrap());
    Ok(())
}

Full example.

Select a quantization with --quant

Use --quant <level> to request a quantization level. Numeric levels (2, 3, 4, 5, 6, 8) can be used with standard and GGUF model sources. Standard model repositories also accept ISQ (in-situ quantization) names such as q4k and afq8; GGUF repositories accept supported Q/K artifact names such as q4k. See GGUF compatibility for the available storage formats. For a GGUF repository, --quant selects a matching file already present in the repository. Other Hugging Face repositories load a matching prebuilt UQFF (Universal Quantized File Format) from mistralrs-community/<model-name>-UQFF when available and otherwise apply ISQ.

mistralrs run --quant 4 -m Qwen/Qwen3-4B

# Select a published 4-bit GGUF without naming the file.
mistralrs run -m unsloth/Qwen3.5-4B-GGUF --quant 4

For standard model repositories, --quant auto probes your hardware (the same analysis as mistralrs tune) and picks a level, or runs at full precision if the model fits. Choose an explicit supported bit width or artifact name for a GGUF repository. --quant conflicts with the explicit knobs --isq and --from-uqff; use those when you want to force ISQ or a specific UQFF file. Choosing a level and the full set of quantization options are covered in the quantization guide.

Local model directories

-m accepts a local path to a directory containing the model files (safetensors plus configs, or a Mistral-native consolidated.safetensors layout):

mistralrs run -m /path/to/model-dir

Local model directories are read straight from disk. For safetensors and Mistral-native directories, --quant goes directly to ISQ. A GGUF-only directory instead selects a matching GGUF. If one directory mixes GGUF with safetensors, UQFF, or other model weights, add --format gguf to select from its GGUF files. A local multimodal GGUF may fetch missing configuration or processor assets from its original model repository; set HF_HUB_OFFLINE=1 to require cache-only loading.

Run GGUF models

GGUF files can be used with interactive chat and the OpenAI-compatible server:

# Run an exact local file.
mistralrs run -f /path/to/model.gguf

# Select and serve a published 4-bit GGUF.
mistralrs serve -m unsloth/Qwen3.5-4B-GGUF --quant 4

The .gguf filename selects the format. mistral.rs uses embedded tokenizer and chat-template metadata when present, and discovers external configuration and projector files when the repository identifies them unambiguously. Use --tok-model-id and --mmproj to override those selections. The same model flags work with run, serve, and bench.

See Run GGUF models for exact Hub files, automatic quantization selection, multimodal input, LoRA, ISQ, offline loading, and Python and Rust examples. The GGUF support reference lists compatible architectures, quantization formats, and feature combinations.

Forcing an architecture

Auto-detection covers normal checkpoints. For text models with a non-standard config, --arch (Python: arch=Architecture...) forces the loader; the accepted names are the lowercase forms in the supported models reference. Multimodal, speech, embedding, and diffusion architectures are always auto-detected on the CLI.

Chat template overrides

Some repos ship a missing or broken chat template. Pass -c/--chat-template <file> (a .json or .jinja file) or --jinja-explicit <file> to override it; bundled fixes live in the repo’s chat_templates/ directory. See chat templates for symptoms and how to write your own.

Running offline

Set HF_HUB_OFFLINE=1 to guarantee no network calls are made to the Hugging Face Hub. Files and repo listings are then served from the local cache only, and missing files fail fast instead of hanging on a download.

# on a machine with network access: populate the cache
mistralrs run -m Qwen/Qwen3-4B

# later, or on the air-gapped machine with the cache copied over
HF_HUB_OFFLINE=1 mistralrs serve -m Qwen/Qwen3-4B

Pre-download with huggingface-cli download <repo> or by running mistral.rs once online; mistralrs cache list shows what is cached. Files resolve from $HF_HUB_CACHE, falling back to $HF_HOME/hub, then ~/.cache/huggingface/hub. A complete local model directory reads from disk. For a local multimodal GGUF whose supporting assets are not stored beside it, cache those assets or point --tok-model-id at a local asset directory before running offline. Related variables (HF_HOME, HF_TOKEN, …) are in the environment variables reference.

Model-specific behavior

Most models need nothing beyond -m. The exceptions (thinking tags, MoE (Mixture of Experts) quantization, template fixes, MatFormer slices) are collected in model family notes.