Expand description
This will be the API as of v0.4.0. Other APIs will not be deprecated, but moved into a module such as this one.
Structs§
- Configure a text model with the various parameters for loading, running, and other inference behaviors.
- Wrapper of
GgufModelBuilderfor LoRA models. - Configure a text GGUF model with the various parameters for loading, running, and other inference behaviors.
- Wrapper of
GgufModelBuilderfor X-LoRA models. - Wrapper of
TextModelBuilderfor LoRA models. - The object used to interact with the model. This can be used with many varietes of models,
and as such may be created with one of: - Builder for PagedAttention metadata.
- A way to add messages with finer control given.
- Plain text (chat) messages.
- Configure a text model with the various parameters for loading, running, and other inference behaviors.
- Configure a UQFF text model with the various parameters for loading, running, and other inference behaviors. This wraps and implements
DerefMutfor the TextModelBuilder, so users should take care to not call UQFF-related methods. - Configure a UQFF text model with the various parameters for loading, running, and other inference behaviors. This wraps and implements
DerefMutfor the VisionModelBuilder, so users should take care to not call UQFF-related methods. - Text (chat) messages with images.
- Configure a vision model with the various parameters for loading, running, and other inference behaviors.
- Wrapper of
TextModelBuilderfor X-LoRA models.
Enums§
- A chat message role.
Traits§
- A type which can be used as a chat request.
Functions§
- Gets the best device, cpu, cuda if compiled with CUDA, or Metal