Quantize a model
Quantization stores a model’s weights at lower precision so it needs less memory. A 14B model in BF16 (2 bytes per parameter) needs about 28 GB for its weights; at 4 bits it needs about 7 GB.
To quantize, pass a bit width: 2, 3, 4, 5, 6, or 8. Lower is smaller and loses more precision. You can also pass auto to let mistral.rs pick, or a format name like q4k or fp8.
mistralrs run -m Qwen/Qwen3-4B --quant 4--quant works the same way with run, serve, and bench, and as the quant key in a TOML config.
from mistralrs import Runner, Which
runner = Runner( which=Which.Plain(model_id="Qwen/Qwen3-4B"), in_situ_quant="4",)in_situ_quant takes a bit width or a format name such as "q4k". Full example.
use mistralrs::{IsqBits, ModelBuilder};
let model = ModelBuilder::new("Qwen/Qwen3-4B") .with_auto_isq(IsqBits::Four) .build() .await?;with_auto_isq picks the format that suits your hardware at that bit width. To ask for an exact format, use .with_isq(IsqType::Q4K). Full example.
How mistral.rs chooses what to load
Section titled “How mistral.rs chooses what to load”On the CLI, --quant uses a copy of the model that’s already quantized when one exists, so you skip the conversion and download less. What happens depends on what -m points to:
-m points to |
What --quant 4 does |
|---|---|
| A Hugging Face model with safetensors weights | Looks for a prebuilt UQFF in mistralrs-community/<name>-UQFF. For Qwen/Qwen3-4B that’s mistralrs-community/Qwen3-4B-UQFF. If that repo has a 4-bit file, mistral.rs downloads and loads it. If not, it downloads the original weights and quantizes them while loading. |
| A local safetensors directory | Quantizes the weights while loading. |
| A GGUF repo or a local directory of GGUF files | Picks the matching GGUF file. For 4 it tries Q4_K_M, then Q4_K_S, Q4_K_L, Q4_1, and Q4_0. You can also pass the file’s own label, like --quant q5_k_s. IQ files aren’t supported. |
| A UQFF repo or directory | Loads its 4-bit UQFF file. If it has none, loading fails and lists the files it does have. |
A bit width matches any format at that width, preferring AFQ on Metal and GGML formats (q4k, q8_0) elsewhere. A format name like q4k matches only that format. The startup log says which file or format was picked.
To skip the UQFF lookup and always quantize the original weights, use --isq instead of --quant. With GGUF, --isq requantizes a file you name with -f:
mistralrs serve -m Qwen/Qwen3-4B --isq 4mistralrs serve -f /path/model-BF16.gguf --isq q4kThe Python and Rust SDKs always quantize while loading and never look for a prebuilt UQFF. To load one from the SDKs, pass it explicitly as shown in the UQFF guide.
Quantizing while loading is called ISQ (in-situ quantization). mistral.rs quantizes each layer as it reads it, so the full-precision model never has to fit in memory at once, but every load takes longer than loading a prebuilt file. To pay that cost only once, save the result as UQFF.
How much memory you save
Section titled “How much memory you save”Footprint scales roughly linearly with bits per weight: a BF16 model uses about half the memory at --quant 8 and a quarter at --quant 4. The KV cache is a separate budget that depends on context length, not on weight quantization. Use nvidia-smi (or equivalent) to measure, and mistralrs tune to estimate before downloading anything.
Pick a bit width
Section titled “Pick a bit width”Start with 8 if it fits and 4 if it doesn’t.
| Bits | Quality label in tune |
Use it when |
|---|---|---|
8 |
Near-lossless | The model fits at 8 bits. Output stays closest to the original. |
6, 5 |
Good | 8-bit is a little too big. |
4 |
Acceptable | You need a large model to fit. This is the most common choice. |
3, 2 |
Degraded | Nothing larger fits. Quality drops the most here, so add an imatrix to limit the loss. |
Each width maps to a format suited to your hardware. On Metal it’s AFQ (4 is AFQ4). On CUDA and CPU it’s a GGML format (4 is Q4K, 8 is Q8_0). 5 is Q5K everywhere. The full mapping is in the numeric shorthands table.
Pick a specific format
Section titled “Pick a specific format”--quant and --isq also take format names:
mistralrs run -m Qwen/Qwen3-4B --quant q4k # 4-bit K-quantmistralrs run -m Qwen/Qwen3-4B --quant afq4 # 4-bit AFQ, built for Apple Siliconmistralrs run -m Qwen/Qwen3-4B --quant q8_0 # 8-bit block format| Names | What they are |
|---|---|
q2k, q3k, q4k, q5k, q6k, q8k |
GGML K-quants. They run on every backend and are the only formats that use an imatrix. q8k isn’t available in CUDA builds. |
afq2, afq3, afq4, afq6, afq8 |
Affine quantization with native Apple Silicon kernels. Also runs on CUDA and CPU. |
q4_0, q4_1, q5_0, q5_1, q8_0, q8_1 |
Older GGML block formats. q8_1 isn’t available in CUDA builds. |
fp8 |
8-bit floating point (E4M3) weights. |
mxfp4 |
4-bit microscaling. CUDA and Metal only. |
hqq4, hqq8 |
Half-quadratic quantization at 4 or 8 bits. |
f8q8 |
8-bit format for CPU only. |
If you pick a format your hardware can’t run, loading fails with an error. Hardware details are in the quantization types reference. To give some layers a different format from the rest, use a topology file.
Let mistral.rs decide
Section titled “Let mistral.rs decide”mistralrs run -m Qwen/Qwen3-4B --quant auto--quant auto checks your hardware and picks the highest bit width that fits, trying 6, 5, 4, and then 3 bits (6, 4, and 3 on Metal). If none of those fit it tries 2 bits, and it fails if nothing fits. It then loads that width the same way an explicit --quant does, prebuilt UQFF first.
auto always quantizes. To run at full precision, leave out --quant. It doesn’t pick GGUF files, so pass a bit width or label for GGUF repos.
Estimate with mistralrs tune
Section titled “Estimate with mistralrs tune”mistralrs tune runs the same check and prints the result instead of loading the model. It reads only the model’s config files, so nothing large is downloaded:
mistralrs tune -m Qwen/Qwen3-4BYou get one row per format with its estimated size, share of GPU memory, how much context fits beside it, the quality label from the bit width table, and a status: Recommended, Fits, Hybrid (split across GPU and CPU), or Too Large. A ready-to-run command for the recommended row is printed below the table.
--profile changes which widths tune prefers. It recommends the first one in the list that fits:
--profile |
CUDA and CPU | Metal |
|---|---|---|
quality |
8, 6, 5, 4, 3, 2 bits | 8, 6, 4, 3 bits |
balanced (default, and what --quant auto uses) |
6, 5, 4, 3 bits | 6, 4, 3 bits |
fast |
4, 3, 2 bits | 4, 3, 2 bits |
Other options:
mistralrs tune -m Qwen/Qwen3-4B --isq q4k # prefer one format if it fitsmistralrs tune -m Qwen/Qwen3-4B --json # machine-readable outputmistralrs tune -m Qwen/Qwen3-4B --emit-config qwen.toml # save the recommendation as a configmistralrs from-config -f qwen.toml # then run itAll flags are in the CLI reference.
imatrix
Section titled “imatrix”An importance matrix (imatrix) records which weights matter most when the model runs on sample text, so the quantizer can keep more precision where it counts. It helps most at low bit widths, and only K-quants (q2k to q6k) use it. On Metal, name a K-quant (--isq q4k), because bit widths pick AFQ there.
Pass one of these together with --isq:
| Flag | What it does |
|---|---|
--imatrix <file> |
Uses an existing imatrix: a llama.cpp .imatrix file or a mistral.rs .cimatrix file. |
--calibration-file <file> |
Runs a text file through the model at load time to build the imatrix, then quantizes. This repeats on every load. |
You can’t use both at once. In the SDKs, set imatrix or calibration_file on Which (Python), or call .with_imatrix() or .with_calibration_file() (Rust). Full examples: Rust, Python.
To build an imatrix from a server’s real traffic instead, see online calibration.
Quantize only the experts of an MoE model
Section titled “Quantize only the experts of an MoE model”--isq-organization moqe quantizes only the expert layers of a Mixture of Experts model and keeps the shared layers at their original precision. It’s useful when the experts make up most of the model’s parameters:
mistralrs serve -m microsoft/Phi-3.5-MoE-instruct --isq 4 --isq-organization moqeWithout it (--isq-organization default), every layer mistral.rs can quantize is quantized. The setting only matters when mistral.rs does the quantizing, including when you make a UQFF; it doesn’t change a GGUF or UQFF that’s already quantized. In the SDKs, use organization=IsqOrganization.MoQE (Python) or .with_mixture_qexperts_isq() (Rust). Full example.
Requantize a running model
Section titled “Requantize a running model”If a model was quantized at load, you can switch it to another format without restarting the server:
curl -X POST localhost:1234/re_isq \ -H "Content-Type: application/json" \ -d '{"ggml_type": "Q4K"}'This requantizes the weights already in memory, so moving to a higher bit width doesn’t bring back precision that’s already gone. It returns an error for an unquantized model, or a UQFF or GGUF loaded as is. The SDK equivalents are runner.send_re_isq("Q4K") (Python) and model.re_isq_model(IsqType::Q4K).await? (Rust).
Models that are already quantized
Section titled “Models that are already quantized”These load as they are, with no conversion:
- UQFF: picked up by
--quantwhen a match exists, or loaded by name with--from-uqff. See UQFF. - GGUF: pick a file from a GGUF repo with
--quant, or name one with-f. See GGUF support for compatible models. - GPTQ, AWQ, FP8, and NVFP4 checkpoints: detected from the model’s config, so leave out
--quantand--isq. See the quantization types reference.