Struct Model 

pub struct Model { /* private fields */ }

The object used to interact with the model. This can be used with many varietes of models,
and as such may be created with one of:

• TextModelBuilder
• LoraModelBuilder
• XLoraModelBuilder
• GgufModelBuilder
• GgufLoraModelBuilder
• GgufXLoraModelBuilder
• VisionModelBuilder
• AnyMoeModelBuilder

Implementations

impl Model

pub fn new(runner: Arc<MistralRs>) -> Self

pub async fn stream_chat_request<R: RequestLike>( &self, request: R, ) -> Result<Stream<'_>>

Generate with the model.

pub async fn send_chat_request<R: RequestLike>( &self, request: R, ) -> Result<ChatCompletionResponse>

Generate with the model.

pub async fn send_raw_chat_request<R: RequestLike>( &self, request: R, ) -> Result<(Vec<Tensor>, Vec<u32>)>

Generate with the model, returning raw logits of the first token generated.

Returns the chunks of the logits (1 or more, determined by prompt batchsize) and the tokens.

pub async fn generate_image( &self, prompt: impl ToString, response_format: ImageGenerationResponseFormat, generation_params: DiffusionGenerationParams, ) -> Result<ImageGenerationResponse>

pub async fn generate_speech( &self, prompt: impl ToString, ) -> Result<(Arc<Vec<f32>>, usize, usize)>

Generate audio given a (model specific) prompt.

This returns: (pcm, sampling rate, channels)

pub async fn generate_embeddings( &self, request: EmbeddingRequestBuilder, ) -> Result<Vec<Vec<f32>>>

Generate embeddings for one or more inputs configured via an EmbeddingRequestBuilder.

Returns one embedding vector per input in the same order they were added.

pub async fn generate_embedding( &self, prompt: impl ToString, ) -> Result<Vec<f32>>

Convenience wrapper for generating a single embedding.

pub async fn re_isq_model(&self, isq_type: IsqType) -> Result<()>

Reapply ISQ to the model. This will be done on whatever device the model is already on.

pub async fn tokenize( &self, text: Either<TextMessages, String>, tools: Option<Vec<Tool>>, add_special_tokens: bool, add_generation_prompt: bool, enable_thinking: Option<bool>, ) -> Result<Vec<u32>>

Tokenize some text or messages.

• tools is only used if messages are provided.

pub async fn detokenize( &self, tokens: Vec<u32>, skip_special_tokens: bool, ) -> Result<String>

Detokenize some tokens.

pub fn config(&self) -> Result<MistralRsConfig, String>

Retrieve some information about this model.

pub fn inner(&self) -> &MistralRs