mistralrs_core

Function parse_isq_value

Source 
pub fn parse_isq_value(s: &str) -> Result<IsqType, String>
Expand description

Parse ISQ value.

If the provided value is a valid integer (one of 2,3,4,5,6,8), the best quantization type will be chosen. Note that the fallback is always a Q/K quantization but on Metal 2,3,4,6,8 uses the fast AFQ.

One of:

  • Q4_0
  • Q4_1
  • Q5_0
  • Q5_1
  • Q8_0
  • Q8_1
  • Q2K
  • Q3K
  • Q4K
  • Q5K
  • Q6K
  • Q8K
  • HQQ1
  • HQQ2
  • HQQ3
  • HQQ4
  • HQQ8
  • AFQ2
  • AFQ3
  • AFQ4
  • AFQ6
  • AFQ8