Struct Model

pub struct Model { /* private fields */ }

The object used to interact with the model. This can be used with many varietes of models,
and as such may be created with one of:

• TextModelBuilder
• LoraModelBuilder
• XLoraModelBuilder
• GgufModelBuilder
• GgufLoraModelBuilder
• GgufXLoraModelBuilder
• VisionModelBuilder
• AnyMoeModelBuilder

Implementations

impl Model

pub fn new(runner: Arc<MistralRs>) -> Self

pub async fn stream_chat_request<R: RequestLike>( &self, request: R, ) -> Result<Stream<'_>>

Generate with the model.

pub async fn send_chat_request<R: RequestLike>( &self, request: R, ) -> Result<ChatCompletionResponse>

Generate with the model.

pub async fn send_raw_chat_request<R: RequestLike>( &self, request: R, ) -> Result<(Vec<Tensor>, Vec<u32>)>

Generate with the model, returning raw logits of the first token generated.

Returns the chunks of the logits (1 or more, determined by prompt batchsize) and the tokens.

pub async fn generate_image( &self, prompt: impl ToString, response_format: ImageGenerationResponseFormat, generation_params: DiffusionGenerationParams, ) -> Result<ImageGenerationResponse>

pub async fn activate_adapters<A: ToString>( &self, adapters: Vec<A>, ) -> Result<()>

Activate certain adapters on the model, they will be used for requests which do not specify unique adapters.

pub async fn re_isq_model(&self, isq_type: IsqType) -> Result<()>

Reapply ISQ to the model. This will be done on whatever device the model is already on.

pub async fn tokenize( &self, text: Either<TextMessages, String>, tools: Option<Vec<Tool>>, add_special_tokens: bool, add_generation_prompt: bool, ) -> Result<Vec<u32>>

Tokenize some text or messages.

• tools is only used if messages are provided.

pub async fn detokenize( &self, tokens: Vec<u32>, skip_special_tokens: bool, ) -> Result<String>

Detokenize some tokens.

pub fn config(&self) -> &MistralRsConfig

Retrieve some information about this model.

pub fn inner(&self) -> &MistralRs