Rust SDK getting started
The Rust SDK embeds the engine directly into a Rust program. A Rust toolchain is required; see rustup.rs.
use anyhow::Result;use mistralrs::{IsqBits, ModelBuilder, TextMessageRole, TextMessages};
#[tokio::main]async fn main() -> Result<()> { let model = ModelBuilder::new("Qwen/Qwen3-4B") .with_auto_isq(IsqBits::Four) .with_logging() .build() .await?;
let messages = TextMessages::new().add_message( TextMessageRole::User, "In one sentence, what is Rust known for?", );
let response = model.send_chat_request(messages).await?; println!("{}", response.choices[0].message.content.as_ref().unwrap());
Ok(())}Run with cargo run --release. The first run downloads the weights into the Hugging Face cache.
Full example: text-generation.
Project setup
Section titled “Project setup”cargo new --bin hello-mistralrscd hello-mistralrsAdd the dependencies to Cargo.toml:
[dependencies]anyhow = "1"mistralrs = "0.9.5"tokio = { version = "1", features = ["full"] }The default features build for CPU. For GPU acceleration, enable the matching feature:
# NVIDIA GPU (CUDA)mistralrs = { version = "0.9.5", features = ["cuda", "flash-attn", "cudnn"] }
# Apple Silicon (Metal)mistralrs = { version = "0.9.5", features = ["metal"] }
# Intel CPU with MKLmistralrs = { version = "0.9.5", features = ["mkl"] }Feature names match the CLI build features. The cargo features reference lists every option.
The pieces
Section titled “The pieces”ModelBuilder::new accepts a Hugging Face ID or local path. build() loads it and returns a reusable Model.
with_auto_isq(IsqBits::Four) matches --isq 4 on the CLI: the engine selects an optimal 4-bit format per platform (AFQ4 on Metal, Q4K on CUDA or CPU). To pin a specific format, use with_isq(IsqType::Q4K); see the quantization reference.
TextMessages assembles a basic chat conversation. For sampling controls, tool schemas, or logprobs, use RequestBuilder.
The Rust SDK reference lists the full Model surface; specialized builders (GgufModelBuilder, EmbeddingModelBuilder, LoraModelBuilder, …) are on docs.rs.
Memory and device placement
Section titled “Memory and device placement”The builder automatically chooses device placement and KV cache capacity. PagedAttention is enabled automatically on supported CUDA and Metal devices. Add .with_max_num_seqs(4) to allow up to four concurrent sequences; the default is 32.
To choose a shared KV cache capacity, import mistralrs::MemoryGoal and add .with_memory_goal(MemoryGoal::KvTokens(65536)) to the builder. This capacity is shared across requests; .with_max_model_len(8192) separately limits each request’s prompt plus output. Explicit memory goals require paging. See memory budgets for sizing options and layer mapping for manual placement.
Streaming
Section titled “Streaming”To stream instead of calling send_chat_request, consume incremental text:
use mistralrs::Response;use std::io::Write;
let mut stream = model.stream_chat_request(messages).await?;let mut out = std::io::stdout().lock();while let Some(item) = stream.next().await { match item { Response::Chunk(chunk) => { if let Some(text) = chunk.choices.first().and_then(|c| c.delta.content.as_ref()) { out.write_all(text.as_bytes())?; out.flush()?; } } Response::InternalError(e) | Response::ValidationError(e) => { anyhow::bail!("{e}"); } Response::ModelError(message, _) => anyhow::bail!("{message}"), _ => {} }}The stream also implements futures::Stream. See streaming for tool events, multiple choices, cancellation, and spawning tasks.
ModelBuilder::build() performs all model loading and is expensive. Call it once at startup. Model methods take &self and are safe to call concurrently, so you can share one instance by reference. To move it across spawned tasks, wrap it in Arc<Model> (Model does not implement Clone).
To serve HTTP alongside in-process requests, see embedding in Axum.