Struct CublasLtWrapper

pub struct CublasLtWrapper {}

Implementations

impl CublasLtWrapper

pub fn batch_matmul_f8( &self, a: &Tensor, b: &Tensor, dequant_a_scale: &Tensor, dequant_b_scale: &Tensor, quantize_scale: &Tensor, out: Option<&Tensor>, alpha: Option<f32>, beta: Option<f32>, bias: Option<&Tensor>, act: Option<CandleActivation>, out_dtype: F8MatmulOutType, ) -> Result<Tensor>

Fused batch matmul + add + Relu/Gelu activation using CublasLt for F8 dtypes.

Arguments

• a - Input tensor of size BxMxK
• b - Input tensor of size BxNxK
• dequant_a_scale - F32 scalar tensor, used to a the out tensor.
• dequant_b_scale - F32 scalar tensor, used to b the out tensor.
• quantize_scale - F32 scalar tensor, used to requantize.
• out - Optional Output tensor of size BxNxK. If set and beta != 0, will be added to the end result of A*B before act
• alpha - Optional scaling factor for A*B
• beta - Optional scaling factor for C
• bias - Optional bias tensor of size M
• act - Optional Gelu or Relu activation. If set, will be added to the end result

The resulting tensor is of shape NxM

pub fn batch_matmul( &self, a: &Tensor, b: &Tensor, out: Option<&Tensor>, alpha: Option<f32>, beta: Option<f32>, bias: Option<&Tensor>, act: Option<CandleActivation>, ) -> Result<Tensor>

Fused batch matmul + add + Relu/Gelu activation using CublasLt.

Arguments

• a - Input tensor of size BxMxK
• b - Input tensor of size BxNxK
• out - Optional Output tensor of size BxNxK. If set and beta != 0, will be added to the end result of A*B before act
• alpha - Optional scaling factor for A*B
• beta - Optional scaling factor for C
• bias - Optional bias tensor of size M
• act - Optional Gelu or Relu activation. If set, will be added to the end result

The resulting tensor is of shape NxM

Trait Implementations

impl Clone for CublasLtWrapper

fn clone(&self) -> CublasLtWrapper

Returns a copy of the value.

fn clone_from(&mut self, source: &Self)

Performs copy-assignment from source.

impl Debug for CublasLtWrapper

fn fmt(&self, f: &mut Formatter<'_>) -> Result

Formats the value using the given formatter.