Crate mistralrs

source
Expand description

This crate provides an asynchronous API to mistral.rs.

To get started loading a model, check out the following builders:

Check out the v0_4_api module for concise documentation of this, newer API.

§Example

use anyhow::Result;
use mistralrs::{
    IsqType, PagedAttentionMetaBuilder, TextMessageRole, TextMessages, TextModelBuilder,
};

#[tokio::main]
async fn main() -> Result<()> {
    let model = TextModelBuilder::new("microsoft/Phi-3.5-mini-instruct".to_string())
        .with_isq(IsqType::Q8_0)
        .with_logging()
        .with_paged_attn(|| PagedAttentionMetaBuilder::default().build())?
        .build()
        .await?;

    let messages = TextMessages::new()
        .add_message(
            TextMessageRole::System,
            "You are an AI agent with a specialty in programming.",
        )
        .add_message(
            TextMessageRole::User,
            "Hello! How are you? Please write generic binary search function in Rust.",
        );

    let response = model.send_chat_request(messages).await?;

    println!("{}", response.choices[0].message.content.as_ref().unwrap());
    dbg!(
        response.usage.avg_prompt_tok_per_sec,
        response.usage.avg_compl_tok_per_sec
    );

    Ok(())
}

§Streaming example

use anyhow::Result;
use mistralrs::{
    IsqType, PagedAttentionMetaBuilder, TextMessageRole, TextMessages, TextModelBuilder, Response
};

#[tokio::main]
async fn main() -> Result<()> {
    let model = TextModelBuilder::new("microsoft/Phi-3.5-mini-instruct".to_string())
        .with_isq(IsqType::Q8_0)
        .with_logging()
        .with_paged_attn(|| PagedAttentionMetaBuilder::default().build())?
        .build()
        .await?;

    let messages = TextMessages::new()
        .add_message(
            TextMessageRole::System,
            "You are an AI agent with a specialty in programming.",
        )
        .add_message(
            TextMessageRole::User,
            "Hello! How are you? Please write generic binary search function in Rust.",
        );

    let mut stream = model.stream_chat_request(messages).await?;

    while let Some(chunk) = stream.next().await {
        if let Response::Chunk(chunk) = chunk{
            print!("{}", chunk.choices[0].delta.content);
        }
        // Handle the error cases.

    }
    Ok(())
}

Re-exports§

Modules§

  • layers
  • llguidance
  • v0_4_api
    This will be the API as of v0.4.0. Other APIs will not be deprecated, but moved into a module such as this one.

Structs§

Enums§

Constants§

Statics§

Traits§

  • CustomLogitsProcessor
    Customizable logits processor.
  • Loader
    The Loader trait abstracts the loading process. The primary entrypoint is the load_model method.
  • ModelPaths
    ModelPaths abstracts the mechanism to get all necessary files for running a model. For example LocalModelPaths implements ModelPaths when all files are in the local file system.
  • Pipeline
  • TryIntoDType
    Type which can be converted to a DType
  • VisionPromptPrefixer
    Prepend a vision tag appropriate for the model to the prompt. Image indexing is assumed that start at

Functions§

Type Aliases§