Struct MistralRsBuilder

pub struct MistralRsBuilder { /* private fields */ }

The MistralRsBuilder takes the pipeline and a scheduler method and constructs an Engine and a MistralRs instance. The Engine runs on a separate thread, and the MistralRs instance stays on the calling thread.

Implementations

impl MistralRsBuilder

pub fn new( pipeline: Arc<Mutex<dyn Pipeline>>, method: SchedulerConfig, throughput_logging: bool, search_embedding_model: Option<BertEmbeddingModel>, ) -> Self

Creates a new builder with the given pipeline, scheduler method, logging flag, and optional embedding model for web search. To override the search callback, use .with_search_callback(...) on the builder.

pub fn with_log(self, log: String) -> Self

pub fn with_opt_log(self, log: Option<String>) -> Self

pub fn with_truncate_sequence(self, truncate_sequence: bool) -> Self

pub fn with_no_kv_cache(self, no_kv_cache: bool) -> Self

pub fn with_no_prefix_cache(self, no_prefix_cache: bool) -> Self

pub fn with_prefix_cache_n(self, prefix_cache_n: usize) -> Self

pub fn with_disable_eos_stop(self, disable_eos_stop: bool) -> Self

pub fn with_search_callback(self, search_callback: Arc<SearchCallback>) -> Self

Use a custom callback to gather search results.

pub fn with_tool_callback( self, name: impl Into<String>, tool_callback: Arc<ToolCallback>, ) -> Self

Register a custom callback for the specified tool name.

pub fn build(self) -> Arc<MistralRs>