Paged attention
Paged attention stores the KV cache in small blocks drawn from one shared pool. A request takes blocks as it grows and returns them when it finishes, and requests that start with the same prompt share blocks instead of copying them. This lets many requests run at once without each one reserving a worst-case cache up front.
It’s on by default for supported models on CUDA and Metal, and mistral.rs sizes the pool from whatever memory is left after loading the model. Most people never need to change anything here. Read on if you want to set the cache size yourself or turn paging off.
# Fixed cache size, in bytes or tokensmistralrs serve -m Qwen/Qwen3-4B --kv-cache 8GiBmistralrs serve -m Qwen/Qwen3-4B --kv-cache 65536tokens
# Cap everything mistral.rs allocates at 90% of GPU memory; the cache gets what's leftmistralrs serve -m Qwen/Qwen3-4B --gpu-memory-utilization 0.9
# Turn paging offmistralrs serve -m Qwen/Qwen3-4B --paged-attn offfrom mistralrs import Runner, Which
runner = Runner( which=Which.Plain(model_id="Qwen/Qwen3-4B"), memory_goal="8GiB", # or "65536tokens", or 0.9 for a utilization cap)Pass no_paged_attn=True to turn paging off. Full example.
use mistralrs::{MemoryGoal, ModelBuilder};
let model = ModelBuilder::new("Qwen/Qwen3-4B") .with_memory_goal(MemoryGoal::KvTokens(65536)) .build() .await?;The other goals are MemoryGoal::KvBytes(n) and MemoryGoal::Utilization(0.9). Call .without_paged_attn() to turn paging off. Full example.
On CPU, and for models without paged attention support, mistral.rs uses a regular per-request cache. Setting a cache size there is an error.
Memory budget
Section titled “Memory budget”You have three choices:
| Setting | What happens |
|---|---|
| Nothing (default) | mistral.rs loads the weights, sets aside working memory, keeps headroom for the rest of the system, and gives the remainder to the cache. |
--kv-cache 8GiB or --kv-cache 65536tokens |
The cache gets exactly that much. If it doesn’t fit, loading fails instead of quietly giving you less. A byte budget is the total across all of the model’s GPUs. |
--gpu-memory-utilization 0.9 |
Weights, working memory, and cache together stay under 90% of GPU memory. The cache gets whatever is left under the cap. |
Use --kv-cache or --gpu-memory-utilization, not both.
The default headroom is 5% of GPU memory (at least 1 GiB) on discrete GPUs, and 25% (at most 16 GiB) on unified-memory devices like Apple Silicon. On a 64 GiB Mac, for example, mistral.rs plans to use at most 48 GiB. Other apps using memory at startup shrink that further.
The cache is shared by every request in flight. With --kv-cache 65536tokens and 8 similar-length requests running, each has room for about 8,192 tokens, not 65,536. --max-seqs limits how many requests run at once (the default is 32 for serve and 1 for run), and --max-model-len limits how long any one of them can get; see context length.
When the cache runs out, mistral.rs pauses a running request and recomputes it later, which wastes work. If the mistralrs_paged_preemptions_total metric keeps climbing, lower --max-seqs or give the cache more memory. See throughput tuning for more.
FP8 KV cache
Section titled “FP8 KV cache”--pa-cache-type f8e4m3 stores the cache in FP8, which takes half the memory of BF16, so the same budget holds twice as many tokens. It’s independent of weight quantization: you can combine it with any --quant setting. It isn’t available for models that use MLA, such as DeepSeek V3.
Multiple models
Section titled “Multiple models”When serving several models, mistral.rs splits the memory budget between them in proportion to each model’s --max-seqs. A model’s --kv-cache must fit inside its share, or loading fails. Only one --gpu-memory-utilization value can apply to the whole server. A model with paging off still counts toward the split, but its cache isn’t capped by it.
Context length
Section titled “Context length”The KV cache also limits how long one request can be. See context length to check, raise, or cap it.
FlashAttention and FlashInfer
Section titled “FlashAttention and FlashInfer”Every CUDA build includes FlashInfer kernels for paged decode. Building with the flash-attn feature (compute capability 8.0 and newer) or flash-attn-v3 (Hopper) also speeds up prompt processing. Paged attention works without either. See Cargo features.
CUDA graphs
Section titled “CUDA graphs”On CUDA, mistral.rs records each decode step as a CUDA graph and replays it for later tokens. This removes most of the CPU cost of launching kernels, which is a large share of each token’s time on small models. Common batch sizes are recorded at startup and new ones as they appear. Graphs don’t change outputs.
Graphs apply only to generating tokens, not to reading the prompt. If recording fails, mistral.rs logs a warning and runs that model without graphs.
They’re on by default. To rule them out while debugging, set MISTRALRS_CUDA_GRAPHS=0. Internals are in the developer guide.