Struct GGMLLoader

pub struct GGMLLoader { /* private fields */ }

A loader for a GGML model.

Implementations

impl GGMLLoader

pub fn new( model_id: Option<String>, config: GGMLSpecificConfig, quantized_model_id: Option<String>, quantized_filename: Option<String>, xlora_model_id: Option<String>, kind: ModelKind, xlora_order: Option<Ordering>, no_kv_cache: bool, chat_template: Option<String>, tokenizer_json: Option<String>, tgt_non_granular_index: Option<usize>, ) -> GGMLLoader

Trait Implementations

impl Loader for GGMLLoader

fn load_model_from_path( &self, paths: &Box<dyn ModelPaths>, _: &dyn TryIntoDType, device: &Device, silent: bool, mapper: DeviceMapMetadata, in_situ_quant: Option<IsqType>, paged_attn_config: Option<PagedAttentionConfig>, ) -> Result<Arc<Mutex<dyn Pipeline + Sync + Send>>, Error>

Load a model from the specified paths. Also initializes DEBUG.

fn load_model_from_hf( &self, revision: Option<String>, token_source: TokenSource, dtype: &dyn TryIntoDType, device: &Device, silent: bool, mapper: DeviceMapMetadata, in_situ_quant: Option<IsqType>, paged_attn_config: Option<PagedAttentionConfig>, ) -> Result<Arc<Mutex<dyn Pipeline + Sync + Send>>, Error>

If revision is None, then it defaults to main. If dtype is None, then it defaults to the model default (usually BF16). If model is not found on HF, will attempt to resolve locally.

fn get_id(&self) -> String

fn get_kind(&self) -> ModelKind