mistralrs quantize
Generate UQFF quantized model file
mistralrs quantize [OPTIONS] [COMMAND]| Option | Default | Description |
|---|---|---|
-m, --model-id <MODEL_ID> | Hugging Face model ID or local model directory; optional when -f names local files | |
-t, --tokenizer <TOKENIZER> | Path to local tokenizer.json file | |
--dtype <DTYPE> | auto | Model data type |
--format <FORMAT> | Input model format: plain (safetensors) or GGUF. Auto-detected from -f when omitted. Possible values: plain, gguf. | |
-f, --quantized-file <QUANTIZED_FILE> | GGUF filename(s), separated by semicolons for multiple files | |
--mmproj <MMPROJ> | GGUF projector override; auto-selected when unambiguous (semicolon-separated for multiple) | |
--tok-model-id <TOK_MODEL_ID> | Optional model ID overriding configuration, tokenizer, and processor assets for a GGUF model | |
--quant <QUANT> | Select an input GGUF artifact by bit width or quant name | |
--isq <IN_SITU_QUANT> | Output UQFF quantization type(s). Multiple values can be comma-separated or specified via repeated —isq flags (e.g., “—isq q4k,q8_0” or “—isq q4k —isq q8_0”) | |
--isq-organization <ISQ_ORGANIZATION> | ISQ organization strategy: default or moqe | |
--imatrix <IMATRIX> | imatrix file for enhanced quantization | |
--calibration-file <CALIBRATION_FILE> | Calibration file for imatrix generation | |
--cpu | false | Force CPU-only execution |
-n, --device-layers <DEVICE_LAYERS> | Device layer mapping (format: ORD:NUM;… e.g., “0:10;1:20”) | |
--topology <TOPOLOGY> | Topology YAML file for device mapping | |
--hf-cache <HF_CACHE> | Custom Hugging Face cache directory | |
--max-seq-len <MAX_SEQ_LEN> | 4096 | Max sequence length for automatic device mapping |
--max-batch-size <MAX_BATCH_SIZE> | 1 | Max batch size for automatic device mapping |
-o, --output <OUTPUT_PATH> | Output path: a .uqff file path (single ISQ) or a directory (auto-names files per ISQ type) | |
--no-readme | false | Skip README.md model card generation (generated by default in directory mode) |
--uqff-base-model <UQFF_BASE_MODEL> | Base model ID for the generated README (skips interactive prompt) | |
--uqff-repo-id <UQFF_REPO_ID> | HF repo ID for the generated README and upload hint (skips interactive prompt) | |
--max-edge <MAX_EDGE> | Maximum edge length for image resizing (aspect ratio preserved) | |
--max-num-images <MAX_NUM_IMAGES> | Maximum number of images per request | |
--max-image-length <MAX_IMAGE_LENGTH> | Maximum image dimension for device mapping |
mistralrs quantize auto
Section titled “mistralrs quantize auto”Auto-detect model type (recommended)
mistralrs quantize auto [OPTIONS] --isq <IN_SITU_QUANT> --output <OUTPUT_PATH>| Option | Default | Description |
|---|---|---|
-m, --model-id <MODEL_ID> | Hugging Face model ID or local model directory; optional when -f names local files | |
-t, --tokenizer <TOKENIZER> | Path to local tokenizer.json file | |
--dtype <DTYPE> | auto | Model data type |
--format <FORMAT> | Input model format: plain (safetensors) or GGUF. Auto-detected from -f when omitted. Possible values: plain, gguf. | |
-f, --quantized-file <QUANTIZED_FILE> | GGUF filename(s), separated by semicolons for multiple files | |
--mmproj <MMPROJ> | GGUF projector override; auto-selected when unambiguous (semicolon-separated for multiple) | |
--tok-model-id <TOK_MODEL_ID> | Optional model ID overriding configuration, tokenizer, and processor assets for a GGUF model | |
--quant <QUANT> | Select an input GGUF artifact by bit width or quant name | |
--isq <IN_SITU_QUANT> | required | Output UQFF quantization type(s). Multiple values can be comma-separated or specified via repeated —isq flags (e.g., “—isq q4k,q8_0” or “—isq q4k —isq q8_0”) |
--isq-organization <ISQ_ORGANIZATION> | ISQ organization strategy: default or moqe | |
--imatrix <IMATRIX> | imatrix file for enhanced quantization | |
--calibration-file <CALIBRATION_FILE> | Calibration file for imatrix generation | |
--cpu | false | Force CPU-only execution |
-n, --device-layers <DEVICE_LAYERS> | Device layer mapping (format: ORD:NUM;… e.g., “0:10;1:20”) | |
--topology <TOPOLOGY> | Topology YAML file for device mapping | |
--hf-cache <HF_CACHE> | Custom Hugging Face cache directory | |
--max-seq-len <MAX_SEQ_LEN> | 4096 | Max sequence length for automatic device mapping |
--max-batch-size <MAX_BATCH_SIZE> | 1 | Max batch size for automatic device mapping |
-o, --output <OUTPUT_PATH> | required | Output path: a .uqff file path (single ISQ) or a directory (auto-names files per ISQ type). Examples: -o model/model-q4k.uqff or -o output/ |
--no-readme | false | Skip README.md model card generation (generated by default in directory mode) |
--uqff-base-model <UQFF_BASE_MODEL> | Base model ID for the generated README (skips interactive prompt) | |
--uqff-repo-id <UQFF_REPO_ID> | HF repo ID for the generated README and upload hint (skips interactive prompt) | |
--max-edge <MAX_EDGE> | Maximum edge length for image resizing (aspect ratio preserved) | |
--max-num-images <MAX_NUM_IMAGES> | Maximum number of images per request | |
--max-image-length <MAX_IMAGE_LENGTH> | Maximum image dimension for device mapping |
mistralrs quantize text
Section titled “mistralrs quantize text”Text generation model with explicit architecture
mistralrs quantize text [OPTIONS] --isq <IN_SITU_QUANT> --output <OUTPUT_PATH>| Option | Default | Description |
|---|---|---|
-m, --model-id <MODEL_ID> | Hugging Face model ID or local model directory; optional when -f names local files | |
-t, --tokenizer <TOKENIZER> | Path to local tokenizer.json file | |
--dtype <DTYPE> | auto | Model data type |
--format <FORMAT> | Input model format: plain (safetensors) or GGUF. Auto-detected from -f when omitted. Possible values: plain, gguf. | |
-f, --quantized-file <QUANTIZED_FILE> | GGUF filename(s), separated by semicolons for multiple files | |
--mmproj <MMPROJ> | GGUF projector override; auto-selected when unambiguous (semicolon-separated for multiple) | |
--tok-model-id <TOK_MODEL_ID> | Optional model ID overriding configuration, tokenizer, and processor assets for a GGUF model | |
--quant <QUANT> | Select an input GGUF artifact by bit width or quant name | |
-a, --arch <ARCH> | Model architecture (required for text models) | |
--isq <IN_SITU_QUANT> | required | Output UQFF quantization type(s). Multiple values can be comma-separated or specified via repeated —isq flags (e.g., “—isq q4k,q8_0” or “—isq q4k —isq q8_0”) |
--isq-organization <ISQ_ORGANIZATION> | ISQ organization strategy: default or moqe | |
--imatrix <IMATRIX> | imatrix file for enhanced quantization | |
--calibration-file <CALIBRATION_FILE> | Calibration file for imatrix generation | |
--cpu | false | Force CPU-only execution |
-n, --device-layers <DEVICE_LAYERS> | Device layer mapping (format: ORD:NUM;… e.g., “0:10;1:20”) | |
--topology <TOPOLOGY> | Topology YAML file for device mapping | |
--hf-cache <HF_CACHE> | Custom Hugging Face cache directory | |
--max-seq-len <MAX_SEQ_LEN> | 4096 | Max sequence length for automatic device mapping |
--max-batch-size <MAX_BATCH_SIZE> | 1 | Max batch size for automatic device mapping |
-o, --output <OUTPUT_PATH> | required | Output path: a .uqff file path (single ISQ) or a directory (auto-names files per ISQ type). Examples: -o model/model-q4k.uqff or -o output/ |
--no-readme | false | Skip README.md model card generation (generated by default in directory mode) |
--uqff-base-model <UQFF_BASE_MODEL> | Base model ID for the generated README (skips interactive prompt) | |
--uqff-repo-id <UQFF_REPO_ID> | HF repo ID for the generated README and upload hint (skips interactive prompt) |
mistralrs quantize multimodal
Section titled “mistralrs quantize multimodal”Multimodal model
mistralrs quantize multimodal [OPTIONS] --isq <IN_SITU_QUANT> --output <OUTPUT_PATH>| Option | Default | Description |
|---|---|---|
-m, --model-id <MODEL_ID> | Hugging Face model ID or local model directory; optional when -f names local files | |
-t, --tokenizer <TOKENIZER> | Path to local tokenizer.json file | |
--dtype <DTYPE> | auto | Model data type |
--format <FORMAT> | Input model format: plain (safetensors) or GGUF. Auto-detected from -f when omitted. Possible values: plain, gguf. | |
-f, --quantized-file <QUANTIZED_FILE> | GGUF filename(s), separated by semicolons for multiple files | |
--mmproj <MMPROJ> | GGUF projector override; auto-selected when unambiguous (semicolon-separated for multiple) | |
--tok-model-id <TOK_MODEL_ID> | Optional model ID overriding configuration, tokenizer, and processor assets for a GGUF model | |
--quant <QUANT> | Select an input GGUF artifact by bit width or quant name | |
--isq <IN_SITU_QUANT> | required | Output UQFF quantization type(s). Multiple values can be comma-separated or specified via repeated —isq flags (e.g., “—isq q4k,q8_0” or “—isq q4k —isq q8_0”) |
--isq-organization <ISQ_ORGANIZATION> | ISQ organization strategy: default or moqe | |
--imatrix <IMATRIX> | imatrix file for enhanced quantization | |
--calibration-file <CALIBRATION_FILE> | Calibration file for imatrix generation | |
--cpu | false | Force CPU-only execution |
-n, --device-layers <DEVICE_LAYERS> | Device layer mapping (format: ORD:NUM;… e.g., “0:10;1:20”) | |
--topology <TOPOLOGY> | Topology YAML file for device mapping | |
--hf-cache <HF_CACHE> | Custom Hugging Face cache directory | |
--max-seq-len <MAX_SEQ_LEN> | 4096 | Max sequence length for automatic device mapping |
--max-batch-size <MAX_BATCH_SIZE> | 1 | Max batch size for automatic device mapping |
-o, --output <OUTPUT_PATH> | required | Output path: a .uqff file path (single ISQ) or a directory (auto-names files per ISQ type). Examples: -o model/model-q4k.uqff or -o output/ |
--no-readme | false | Skip README.md model card generation (generated by default in directory mode) |
--uqff-base-model <UQFF_BASE_MODEL> | Base model ID for the generated README (skips interactive prompt) | |
--uqff-repo-id <UQFF_REPO_ID> | HF repo ID for the generated README and upload hint (skips interactive prompt) | |
--max-edge <MAX_EDGE> | Maximum edge length for image resizing (aspect ratio preserved) | |
--max-num-images <MAX_NUM_IMAGES> | Maximum number of images per request | |
--max-image-length <MAX_IMAGE_LENGTH> | Maximum image dimension for device mapping |
mistralrs quantize embedding
Section titled “mistralrs quantize embedding”Embedding model
mistralrs quantize embedding [OPTIONS] --model-id <MODEL_ID> --isq <IN_SITU_QUANT> --output <OUTPUT_PATH>| Option | Default | Description |
|---|---|---|
-m, --model-id <MODEL_ID> | required | Hugging Face model ID or local model directory |
-t, --tokenizer <TOKENIZER> | Path to local tokenizer.json file | |
--dtype <DTYPE> | auto | Model data type |
--isq <IN_SITU_QUANT> | required | Output UQFF quantization type(s). Multiple values can be comma-separated or specified via repeated —isq flags (e.g., “—isq q4k,q8_0” or “—isq q4k —isq q8_0”) |
--isq-organization <ISQ_ORGANIZATION> | ISQ organization strategy: default or moqe | |
--imatrix <IMATRIX> | imatrix file for enhanced quantization | |
--calibration-file <CALIBRATION_FILE> | Calibration file for imatrix generation | |
--cpu | false | Force CPU-only execution |
-n, --device-layers <DEVICE_LAYERS> | Device layer mapping (format: ORD:NUM;… e.g., “0:10;1:20”) | |
--topology <TOPOLOGY> | Topology YAML file for device mapping | |
--hf-cache <HF_CACHE> | Custom Hugging Face cache directory | |
--max-seq-len <MAX_SEQ_LEN> | 4096 | Max sequence length for automatic device mapping |
--max-batch-size <MAX_BATCH_SIZE> | 1 | Max batch size for automatic device mapping |
-o, --output <OUTPUT_PATH> | required | Output path: a .uqff file path (single ISQ) or a directory (auto-names files per ISQ type). Examples: -o model/model-q4k.uqff or -o output/ |
--no-readme | false | Skip README.md model card generation (generated by default in directory mode) |
--uqff-base-model <UQFF_BASE_MODEL> | Base model ID for the generated README (skips interactive prompt) | |
--uqff-repo-id <UQFF_REPO_ID> | HF repo ID for the generated README and upload hint (skips interactive prompt) |