Struct GgufMatMul

pub struct GgufMatMul { /* private fields */ }

Trait Implementations

impl Debug for GgufMatMul

fn fmt(&self, f: &mut Formatter<'_>) -> Result

Formats the value using the given formatter.

impl QuantMethod for GgufMatMul

fn new(method: QuantMethodConfig) -> Result<Self>

where Self: Sized,

fn dequantize_w(&self, out_ty: DType) -> Result<Tensor>

fn forward(&self, a: &Tensor) -> Result<Tensor>

Compute matmul of self and a. self should contain the weights.

fn forward_via_half(&self, a: &Tensor) -> Result<Tensor>

Compute matmul of self and a. self should contain the weights. This may go via half precision if it is supported.

fn quantized_act_type(&self) -> Option<DType>

If a quantized method, return the activation dtype.

fn to_device(&self, dev: &Device) -> Result<Arc<dyn QuantMethod>>

Cast this layer to the given device.

fn size_in_bytes(&self) -> Result<usize>

fn device(&self) -> Device

fn forward_autocast(&self, a: &Tensor) -> Result<Tensor>

Compute matmul of self and a. self should contain the weights. Automatically cast to required quantization actiation type and back