Quantize a model

Quantization stores weights at lower precision so a model fits in less memory. A 14B model in BF16 (2 bytes/param) needs about 28 GB for weights; at 4 bits the same model is about 7 GB. For run, serve, and bench, --quant selects an available pre-quantized artifact. With a safetensors source, it falls back to runtime quantization when no matching UQFF is available:

mistralrs run --quant 4 -m google/gemma-4-E4B-it

--quant 4 first looks for a matching prebuilt UQFF (Universal Quantized File Format) in mistralrs-community/<model>-UQFF and loads it directly if one is published. Otherwise it falls back to ISQ (in-situ quantization): the engine quantizes weights on the fly as they load, picking a hardware-appropriate format (AFQ4 on Metal, Q4K elsewhere). With ISQ the full unquantized model is never resident in memory; loading just takes longer than a pre-quantized file.

For a local safetensors model path, --quant skips the UQFF probe and applies ISQ directly. A local directory containing GGUF files instead selects a matching GGUF artifact.

How much memory you save

Footprint scales roughly linearly with bits per weight: a BF16 model uses about half the memory at --quant 8 and a quarter at --quant 4. The KV cache is a separate budget that depends on context length, not on weight quantization. Use nvidia-smi (or equivalent) to measure, and mistralrs tune to estimate before downloading anything.

Picking a bit width

Supported widths are 2, 3, 4, 5, 6, and 8. Lower bit widths reduce weight memory and generally increase quantization error:

8-bit formats preserve the most weight precision in this set.
5- and 6-bit formats trade some additional precision for lower memory use.
4-bit formats reduce memory further and are commonly used for larger models.
2- and 3-bit formats are the most aggressive options.

An importance matrix can reduce quality loss for K-quant formats at low bit widths. Evaluate the selected format on the target model and workload; bit width alone does not determine output quality.

Picking a specific format

--quant also accepts format names:

mistralrs run --quant q4k -m google/gemma-4-E4B-it     # 4-bit K-quant
mistralrs run --quant afq4 -m google/gemma-4-E4B-it    # AFQ, Metal-optimized
mistralrs run --quant q8_0 -m google/gemma-4-E4B-it    # 8-bit block quant

How the numeric shorthands resolve per backend, plus the full type list and hardware constraints, is in the quantization types reference.

For run, serve, and bench, use --isq instead of --quant only when you want to force runtime ISQ and skip the UQFF lookup. For GGUF, --quant selects an existing variant while --isq requantizes an exact file selected with -f.

Letting mistral.rs decide

--quant auto estimates what fits on the current host and picks for you:

mistralrs run --quant auto -m google/gemma-4-E4B-it

It runs the same analysis as mistralrs tune with the balanced profile. If the model fits at full precision, no quantization is applied; otherwise the recommended type is resolved like an explicit --quant value (prebuilt UQFF preferred, ISQ fallback).

--quant auto does not select a GGUF file. For a GGUF repository, pass an explicit bit width or format name.

Estimate with mistralrs tune

mistralrs tune prints the full analysis instead of acting on it:

mistralrs tune -m google/gemma-4-E4B-it

Output is a table with columns Quant | Est. Size | VRAM % | Context Room | Quality | Status. One row is marked Recommended; the others are marked Fits, Hybrid (split across GPU and CPU), or Too Large. A recommended command line is printed below the table.

tune is a config-based estimator, not a benchmark. It downloads only the model’s config files, computes per-quantization sizes from the architecture, and checks them against detected VRAM. No weights are downloaded, no model is loaded, and the quality column is a fixed tier per format (Baseline, Near-lossless, Good, Acceptable, Degraded), not a measurement.

Variations:

mistralrs tune --profile quality -m google/gemma-4-E4B-it     # quality, balanced (default), fast
mistralrs tune --isq q4k -m google/gemma-4-E4B-it             # bias toward a specific target
mistralrs tune --json -m google/gemma-4-E4B-it                # machine-readable output
mistralrs tune --emit-config gemma.toml -m google/gemma-4-E4B-it

--emit-config writes a TOML config with the recommended settings; run it with mistralrs from-config -f gemma.toml. tune rejects --quant auto since tune is the recommender. All flags: CLI reference.

Quantization on each surface

Use --quant for artifact selection with an ISQ fallback, or --isq to force runtime ISQ:

mistralrs serve -m google/gemma-4-E4B-it --quant 4
mistralrs serve -m google/gemma-4-E4B-it --isq q4k

# Requantize an exact GGUF source instead of selecting a published variant.
mistralrs serve -f /path/model-BF16.gguf --isq q4k

To quantize once and reuse the result, write a UQFF file with mistralrs quantize. See the UQFF guide. The command accepts safetensors sources and compatible GGUF inputs selected with either -f or -m <repo> --quant <level-or-name>.

There is no quantize-at-load HTTP request; a server quantizes via the --quant/--isq flags on the CLI tab. To re-quantize a model already running, POST /re_isq changes its ISQ type without restarting the server. The model must have been loaded with ISQ; models loaded without ISQ are not supported:

curl -X POST localhost:1234/re_isq \
  -H "Content-Type: application/json" \
  -d '{"ggml_type": "Q4K"}'

in_situ_quant accepts the same values as --isq, including numeric shorthands resolved against the actual device:

from mistralrs import Runner, Which

runner = Runner(
    which=Which.Plain(model_id="microsoft/Phi-3.5-mini-instruct"),
    in_situ_quant="Q4K",
)

Use Which.GGUF(...) with the same in_situ_quant argument to requantize a compatible GGUF. Which.GGUF also accepts imatrix, calibration_file, organization, and write_uqff.

Full example

with_auto_isq picks the platform-preferred format at a bit width; with_isq requests an exact type:

use mistralrs::{IsqBits, ModelBuilder};

let model = ModelBuilder::new("Qwen/Qwen3-4B")
    .with_auto_isq(IsqBits::Four)
    .build()
    .await?;

GgufModelBuilder exposes the same ISQ, imatrix, calibration, MoE organization, and UQFF-writing options for compatible GGUF sources.

Full example

What ISQ does

ISQ converts the model during loading rather than requiring a prebuilt artifact. It adds load time but avoids keeping the complete source model in memory at once. To skip the conversion on repeated loads, save the result as UQFF.

Format families

Q*K (q2k-q6k): GGML-compatible block quantization. Broadly applicable, works on all backends.
AFQ (afq2-afq8): affine quantization optimized for Apple Silicon. Runs on Metal (native kernels), CUDA (dedicated backend), and CPU (fallback).
Legacy GGML (q4_0, q4_1, q5_0, q5_1, q8_0): supported for GGUF compatibility.
FP8 (fp8): native FP8 matmul on NVIDIA compute capability 8.9+.
F8Q8 (f8q8): CPU-only 8-bit format.
MXFP4 (4-bit microscaling): CUDA and Metal; CPU is not supported.
HQQ (hqq4, hqq8): alternative 4- and 8-bit schemes.

The numeric shorthand picks a format the active device supports; explicit names override that, and incompatible combinations are rejected at load time. Full constraint table: quantization types reference.

Organization: default vs moqe

--isq-organization selects which layers get quantized:

default: every linear layer the pipeline exposes for quantization.
moqe (MoQE, Mixture of Quantization Experts): only MoE (Mixture of Experts) expert layers; the shared (non-expert) trunk stays at its source precision.

moqe is useful on MoE models where the experts dominate parameter count. It applies when mistral.rs performs ISQ, including UQFF generation. Loading a pre-quantized GGUF or UQFF with --quant does not change that artifact’s layer organization.

imatrix

An importance matrix (imatrix) is a per-column weight derived from running the model on calibration data and accumulating squared input activations. The quantizer uses it to allocate precision to higher-impact weights, which matters most at low bit widths.

Two flags, used with --isq:

--imatrix <path>: load an existing imatrix. Accepts llama.cpp .imatrix files (layer names are mapped automatically) or mistral.rs .cimatrix files.
--calibration-file <path>: generate the importance data at load time by running the calibration text through the model, then quantize.

The two conflict. --imatrix is reused across runs; --calibration-file re-generates on every load. Importance weighting applies to the K-quant formats (q2k-q6k); other formats quantize without it. Calibration runs on all pipelines (text, multimodal, embedding); the calibration text drives the language model, so vision/audio encoder layers quantize without importance data.

To collect an imatrix from a live server’s real traffic instead of a static calibration file, see online calibration.

Full example: Rust, Python.

Interaction with paged attention and flash attention

ISQ applies to weights. The KV cache is a separate budget: paged attention manages its memory independently, and --pa-cache-type quantizes the cache itself. Flash attention operates on activations, not weights, and composes with any ISQ format.

Pre-quantized formats

By default, these load directly with no ISQ conversion:

UQFF: the native pre-quantized format. Loaded automatically by --quant when a sibling UQFF repo exists, or explicitly via --from-uqff. See the UQFF guide.
GGUF: load a local file directly with -f <file.gguf>, or select a published variant with -m <gguf-repo> --quant <level>. Add --isq <type> to an exact -f selection only when you intentionally want to requantize it. See GGUF support for compatible models and file formats.
GPTQ, AWQ: detected from the source repo’s config and loaded directly; no --quant or --isq needed.