Struct GgufModelBuilder

pub struct GgufModelBuilder { /* private fields */ }

Configure a text GGUF model with the various parameters for loading, running, and other inference behaviors.

Implementations

impl GgufModelBuilder

pub fn new(model_id: impl ToString, files: Vec<impl ToString>) -> Self

A few defaults are applied here:

• Token source is from the cache (.cache/huggingface/token)
• Maximum number of sequences running is 32
• Number of sequences to hold in prefix cache is 16.

pub fn with_tok_model_id(self, tok_model_id: impl ToString) -> Self

Source the tokenizer and chat template from this model ID (must contain tokenizer.json and tokenizer_config.json).

pub fn with_prompt_batchsize(self, prompt_batchsize: NonZeroUsize) -> Self

Set the prompt batchsize to use for inference.

pub fn with_topology(self, topology: Topology) -> Self

Set the model topology for use during loading. If there is an overlap, the topology type is used over the ISQ type.

pub fn with_chat_template(self, chat_template: impl ToString) -> Self

Literal Jinja chat template OR Path (ending in .json) to one.

pub fn with_tokenizer_json(self, tokenizer_json: impl ToString) -> Self

Path to a discrete tokenizer.json file.

pub fn with_force_cpu(self) -> Self

Force usage of the CPU device. Do not use PagedAttention with this.

pub fn with_token_source(self, token_source: TokenSource) -> Self

Source of the Hugging Face token.

pub fn with_hf_revision(self, revision: impl ToString) -> Self

Set the revision to use for a Hugging Face remote model.

pub fn with_paged_attn( self, paged_attn_cfg: impl FnOnce() -> Result<PagedAttentionConfig>, ) -> Result<Self>

Enable PagedAttention. Configure PagedAttention with a PagedAttentionConfig object, which can be created with sensible values with a PagedAttentionMetaBuilder.

If PagedAttention is not supported (query with paged_attn_supported), this will do nothing.

pub fn with_max_num_seqs(self, max_num_seqs: usize) -> Self

Set the maximum number of sequences which can be run at once.

pub fn with_no_kv_cache(self) -> Self

Disable KV cache. Trade performance for memory usage.

pub fn with_prefix_cache_n(self, n_seqs: Option<usize>) -> Self

Set the number of sequences to hold in the prefix cache. Set to None to disable the prefix cacher.

pub fn with_logging(self) -> Self

Enable logging.

pub fn with_device_mapping(self, device_mapping: DeviceMapMetadata) -> Self

Provide metadata to initialize the device mapper. Generally, it is more programmatic and easier to use the Topology, see Self::with_topology.

pub async fn build(self) -> Result<Model>