Struct GGUFLoader

pub struct GGUFLoader { /* private fields */ }

Loader for a GGUF model.

Implementations

impl GGUFLoader

pub fn new( model_id: Option<String>, quantized_model_id: String, quantized_filenames: Vec<String>, xlora_model_id: Option<String>, kind: ModelKind, xlora_order: Option<Ordering>, no_kv_cache: bool, chat_template: Option<String>, tgt_non_granular_index: Option<usize>, config: GGUFSpecificConfig, jinja_explicit: Option<String>, ) -> GGUFLoader

Trait Implementations

impl Loader for GGUFLoader

fn load_model_from_hf( &self, revision: Option<String>, token_source: TokenSource, dtype: &dyn TryIntoDType, device: &Device, silent: bool, mapper: DeviceMapSetting, in_situ_quant: Option<IsqType>, paged_attn_config: Option<PagedAttentionConfig>, ) -> Result<Arc<Mutex<dyn Pipeline + Send + Sync>>, Error>

If revision is None, then it defaults to main. If dtype is None, then it defaults to the model default (usually BF16). If model is not found on HF, will attempt to resolve locally.

fn load_model_from_path( &self, paths: &Box<dyn ModelPaths>, dtype: &dyn TryIntoDType, device: &Device, silent: bool, mapper: DeviceMapSetting, in_situ_quant: Option<IsqType>, paged_attn_config: Option<PagedAttentionConfig>, ) -> Result<Arc<Mutex<dyn Pipeline + Send + Sync>>, Error>

Load a model from the specified paths. Also initializes DEBUG.

fn get_id(&self) -> String

fn get_kind(&self) -> ModelKind