Online calibration

Online calibration observes the activations of a live model quantized with ISQ (in-situ quantization), builds an importance matrix from that traffic, and requantizes the tracked layers. The layers are replaced without restarting the server. The model continues serving while statistics are collected; requests received during the apply step wait until it finishes.

Usage

Serve any model with ISQ:

mistralrs serve -m <model> --isq q4k

Online calibration also accepts an exact compatible GGUF source:

mistralrs serve -f /path/model-BF16.gguf --isq q4k

Then drive the lifecycle on the surface of your choice. There is no CLI command for the lifecycle itself; it is driven over HTTP or from an SDK against the running server.

HTTP

# begin observing live traffic; collection adds some decode overhead while on
curl -X POST localhost:1234/calibration/start

# check per-layer collection progress
curl localhost:1234/calibration/status

# requantize from the source weights with the collected statistics and hot-swap
curl -X POST localhost:1234/calibration/apply \
  -H "Content-Type: application/json" \
  -d '{"save_cimatrix": "traffic.cimatrix"}'

status reports how many layers are collecting and the token rows seen per layer. apply harvests the statistics, requantizes, and returns the pre-apply status. The optional save_cimatrix writes the collected importance matrix for reuse with --imatrix.

Rust

The same lifecycle is exposed on Model:

model.begin_calibration().await?;
// ... serve traffic ...
let status = model.calibration_status().await?;
model.apply_calibration(Some("traffic.cimatrix".into())).await?;

Each method has a _with_model variant for multi-model setups. See the full example.

Python

runner.begin_calibration()
# ... serve traffic ...
status = runner.calibration_status()
runner.apply_calibration(save_cimatrix="traffic.cimatrix")

calibration_status returns a CalibrationStatus with collecting, layers, layers_tracking, total_rows, min_rows, and max_rows fields. See the full example.

Activation collection is inactive until start and stops after apply.

Requirements and behavior

• The model must have been loaded with --isq from a supported safetensors or GGUF source; start errors otherwise, including models loaded --from-uqff.
• Importance weighting applies to the K-quant types (Q2K-Q6K). Other Q types and AFQ collect and requantize without using importance weights. HQQ, FP8, F8Q8, and MXFP4 do not support collection, so start returns an error.
• Safetensors layers are reloaded from the source checkpoint when possible. GGUF layers are reloaded from the selected GGUF artifact. Layers that cannot be reloaded fall back to their resident quantized weights; apply logs the fallback count.
• Requantizing an already quantized GGUF or a resident quantized layer can compound quantization loss. Use a high-precision source when output quality is the priority.

See also

• imatrix background.