Struct Model 

pub struct Model { /* private fields */ }

The object used to interact with the model. This can be used with many varietes of models,
and as such may be created with one of:

• TextModelBuilder
• LoraModelBuilder
• XLoraModelBuilder
• GgufModelBuilder
• GgufLoraModelBuilder
• GgufXLoraModelBuilder
• VisionModelBuilder
• AnyMoeModelBuilder

Implementations

impl Model

pub fn new(runner: Arc<MistralRs>) -> Self

pub async fn stream_chat_request<R: RequestLike>( &self, request: R, ) -> Result<Stream<'_>>

Generate with the model (streaming).

pub async fn stream_chat_request_with_model<R: RequestLike>( &self, request: R, model_id: Option<&str>, ) -> Result<Stream<'_>>

Generate with a specific model (streaming). If model_id is None, the request is sent to the default model.

pub async fn send_chat_request<R: RequestLike>( &self, request: R, ) -> Result<ChatCompletionResponse>

Generate with the model (non-streaming).

pub async fn send_chat_request_with_model<R: RequestLike>( &self, request: R, model_id: Option<&str>, ) -> Result<ChatCompletionResponse>

Send a chat request to a specific model. If model_id is None, the request is sent to the default model.

pub async fn send_raw_chat_request<R: RequestLike>( &self, request: R, ) -> Result<(Vec<Tensor>, Vec<u32>)>

Generate with the model, returning raw logits of the first token generated.

Returns the chunks of the logits (1 or more, determined by prompt batchsize) and the tokens.

pub async fn send_raw_chat_request_with_model<R: RequestLike>( &self, request: R, model_id: Option<&str>, ) -> Result<(Vec<Tensor>, Vec<u32>)>

Generate with a specific model, returning raw logits of the first token generated. If model_id is None, the request is sent to the default model.

pub async fn generate_image( &self, prompt: impl ToString, response_format: ImageGenerationResponseFormat, generation_params: DiffusionGenerationParams, ) -> Result<ImageGenerationResponse>

Generate an image using the default model.

pub async fn generate_image_with_model( &self, prompt: impl ToString, response_format: ImageGenerationResponseFormat, generation_params: DiffusionGenerationParams, model_id: Option<&str>, ) -> Result<ImageGenerationResponse>

Generate an image using a specific model. If model_id is None, the request is sent to the default model.

pub async fn generate_speech( &self, prompt: impl ToString, ) -> Result<(Arc<Vec<f32>>, usize, usize)>

Generate audio given a (model specific) prompt.

This returns: (pcm, sampling rate, channels)

pub async fn generate_speech_with_model( &self, prompt: impl ToString, model_id: Option<&str>, ) -> Result<(Arc<Vec<f32>>, usize, usize)>

Generate audio given a (model specific) prompt using a specific model. If model_id is None, the request is sent to the default model.

This returns: (pcm, sampling rate, channels)

pub async fn generate_embeddings( &self, request: EmbeddingRequestBuilder, ) -> Result<Vec<Vec<f32>>>

Generate embeddings for one or more inputs configured via an EmbeddingRequestBuilder.

Returns one embedding vector per input in the same order they were added.

pub async fn generate_embeddings_with_model( &self, request: EmbeddingRequestBuilder, model_id: Option<&str>, ) -> Result<Vec<Vec<f32>>>

Generate embeddings for one or more inputs using a specific model. If model_id is None, the request is sent to the default model.

Returns one embedding vector per input in the same order they were added.

pub async fn generate_embedding( &self, prompt: impl ToString, ) -> Result<Vec<f32>>

Convenience wrapper for generating a single embedding.

pub async fn generate_embedding_with_model( &self, prompt: impl ToString, model_id: Option<&str>, ) -> Result<Vec<f32>>

Convenience wrapper for generating a single embedding using a specific model. If model_id is None, the request is sent to the default model.

pub async fn re_isq_model(&self, isq_type: IsqType) -> Result<()>

Reapply ISQ to the model. This will be done on whatever device the model is already on.

pub async fn re_isq_model_with_model( &self, isq_type: IsqType, model_id: Option<&str>, ) -> Result<()>

Reapply ISQ to a specific model. If model_id is None, the request is sent to the default model.

pub async fn tokenize( &self, text: Either<TextMessages, String>, tools: Option<Vec<Tool>>, add_special_tokens: bool, add_generation_prompt: bool, enable_thinking: Option<bool>, ) -> Result<Vec<u32>>

Tokenize some text or messages.

• tools is only used if messages are provided.

pub async fn tokenize_with_model( &self, text: Either<TextMessages, String>, tools: Option<Vec<Tool>>, add_special_tokens: bool, add_generation_prompt: bool, enable_thinking: Option<bool>, model_id: Option<&str>, ) -> Result<Vec<u32>>

Tokenize some text or messages using a specific model. If model_id is None, the request is sent to the default model.

• tools is only used if messages are provided.

pub async fn detokenize( &self, tokens: Vec<u32>, skip_special_tokens: bool, ) -> Result<String>

Detokenize some tokens.

pub async fn detokenize_with_model( &self, tokens: Vec<u32>, skip_special_tokens: bool, model_id: Option<&str>, ) -> Result<String>

Detokenize some tokens using a specific model. If model_id is None, the request is sent to the default model.

pub fn config(&self) -> Result<MistralRsConfig, String>

Retrieve some information about this model.

pub fn config_with_model( &self, model_id: Option<&str>, ) -> Result<MistralRsConfig, String>

Retrieve some information about a specific model. If model_id is None, returns config for the default model.

pub fn max_sequence_length(&self) -> Result<Option<usize>, MistralRsError>

Returns the maximum supported sequence length for this model, if applicable.

pub fn max_sequence_length_with_model( &self, model_id: Option<&str>, ) -> Result<Option<usize>, MistralRsError>

Returns the maximum supported sequence length for a specific model, if applicable. If model_id is None, returns for the default model.

pub fn list_models(&self) -> Result<Vec<String>, String>

List all available model IDs.

pub fn get_default_model_id(&self) -> Result<Option<String>, String>

Get the current default model ID.

pub fn set_default_model_id(&self, model_id: &str) -> Result<(), String>

Set the default model ID.

pub async fn add_model( &self, model_id: String, pipeline: Arc<Mutex<dyn Pipeline>>, method: SchedulerConfig, config: AddModelConfig, ) -> Result<(), String>

Add a new model dynamically.

pub fn remove_model(&self, model_id: &str) -> Result<(), String>

Remove a model by ID.

pub fn unload_model(&self, model_id: &str) -> Result<(), MistralRsError>

Unload a model from memory (can be reloaded later).

pub async fn reload_model(&self, model_id: &str) -> Result<(), MistralRsError>

Reload a previously unloaded model.

pub fn is_model_loaded(&self, model_id: &str) -> Result<bool, MistralRsError>

Check if a model is currently loaded.

pub fn list_models_with_status( &self, ) -> Result<Vec<(String, ModelStatus)>, MistralRsError>

List all models with their status (Loaded, Unloaded, Reloading).

pub fn inner(&self) -> &MistralRs

Get the underlying MistralRs instance.