Struct GgufModelBuilder 

pub struct GgufModelBuilder { /* private fields */ }

Configure a text GGUF model with the various parameters for loading, running, and other inference behaviors.

Implementations

impl GgufModelBuilder

pub fn new(model_id: impl ToString, files: Vec<impl ToString>) -> Self

A few defaults are applied here:

• Token source is from the cache (.cache/huggingface/token)
• Maximum number of sequences running is 32
• Number of sequences to hold in prefix cache is 16.
• Automatic device mapping with model defaults according to AutoDeviceMapParams
• By default, web searching compatible with the OpenAI web_search_options setting is disabled.

pub fn with_search(self, search_bert_model: BertEmbeddingModel) -> Self

Enable searching compatible with the OpenAI web_search_options setting. This uses the BERT model specified or the default.

pub fn with_search_callback(self, callback: Arc<SearchCallback>) -> Self

Override the search function used when web_search_options is enabled.

pub fn with_tool_callback( self, name: impl Into<String>, callback: Arc<ToolCallback>, ) -> Self

pub fn with_tool_callback_and_tool( self, name: impl Into<String>, callback: Arc<ToolCallback>, tool: Tool, ) -> Self

Register a callback with an associated Tool definition that will be automatically added to requests when tool callbacks are active.

pub fn with_throughput_logging(self) -> Self

Enable runner throughput logging.

pub fn with_jinja_explicit(self, jinja_explicit: String) -> Self

Explicit JINJA chat template file (.jinja) to be used. If specified, this overrides all other chat templates.

pub fn with_tok_model_id(self, tok_model_id: impl ToString) -> Self

Source the tokenizer and chat template from this model ID (must contain tokenizer.json and tokenizer_config.json).

pub fn with_topology(self, topology: Topology) -> Self

Set the model topology for use during loading. If there is an overlap, the topology type is used over the ISQ type.

pub fn with_chat_template(self, chat_template: impl ToString) -> Self

Literal Jinja chat template OR Path (ending in .json) to one.

pub fn with_tokenizer_json(self, tokenizer_json: impl ToString) -> Self

Path to a discrete tokenizer.json file.

pub fn with_force_cpu(self) -> Self

Force usage of the CPU device. Do not use PagedAttention with this.

pub fn with_token_source(self, token_source: TokenSource) -> Self

Source of the Hugging Face token.

pub fn with_hf_revision(self, revision: impl ToString) -> Self

Set the revision to use for a Hugging Face remote model.

pub fn with_paged_attn( self, paged_attn_cfg: impl FnOnce() -> Result<PagedAttentionConfig>, ) -> Result<Self>

Enable PagedAttention. Configure PagedAttention with a PagedAttentionConfig object, which can be created with sensible values with a PagedAttentionMetaBuilder.

If PagedAttention is not supported (query with paged_attn_supported), this will do nothing.

pub fn with_max_num_seqs(self, max_num_seqs: usize) -> Self

Set the maximum number of sequences which can be run at once.

pub fn with_no_kv_cache(self) -> Self

Disable KV cache. Trade performance for memory usage.

pub fn with_prefix_cache_n(self, n_seqs: Option<usize>) -> Self

Set the number of sequences to hold in the prefix cache. Set to None to disable the prefix cacher.

pub fn with_logging(self) -> Self

Enable logging.

pub fn with_device_mapping(self, device_mapping: DeviceMapSetting) -> Self

Provide metadata to initialize the device mapper.

pub fn with_device(self, device: Device) -> Self

Set the main device to load this model onto. Automatic device mapping will be performed starting with this device.

pub async fn build(self) -> Result<Model>