Struct MultiModel

pub struct MultiModel { /* private fields */ }

A simpler multi-model interface that wraps an existing MistralRs instance and provides methods to interact with multiple loaded models.

Implementations

impl MultiModel

pub fn from_model(model: Model) -> Self

Create a MultiModel from an existing Model that has multiple models loaded. This is useful when you’ve created a Model using regular builders and then added more models to it using the add_model method.

pub fn from_mistralrs(mistralrs: Arc<MistralRs>) -> Self

Create a MultiModel directly from a MistralRs instance.

pub fn list_models(&self) -> Result<Vec<String>, String>

List all available model IDs.

pub fn get_default_model_id(&self) -> Result<Option<String>, String>

Get the default model ID.

pub fn set_default_model_id(&self, model_id: &str) -> Result<(), String>

Set the default model ID.

pub fn remove_model(&self, model_id: &str) -> Result<(), String>

Remove a model by ID.

pub async fn add_model( &self, model_id: String, pipeline: Arc<Mutex<dyn Pipeline>>, method: SchedulerConfig, config: AddModelConfig, ) -> Result<(), String>

Add a new model to the multi-model instance.

pub async fn send_chat_request_to_model( &self, request: impl RequestLike, model_id: Option<&str>, ) -> Result<ChatCompletionResponse>

Send a chat request to a specific model.

pub fn inner(&self) -> &MistralRs

Get the underlying MistralRs instance.

pub fn config(&self, model_id: Option<&str>) -> Result<MistralRsConfig, String>

Get configuration for a specific model.