Struct EmbeddingModelBuilder 

pub struct EmbeddingModelBuilder { /* private fields */ }

Configure an embedding model with the various parameters for loading, running, and other inference behaviors.

Implementations

impl EmbeddingModelBuilder

pub fn new(model_id: impl ToString) -> Self

A few defaults are applied here:

• Token source is from the cache (.cache/huggingface/token)
• Automatic device mapping with model defaults according to AutoDeviceMapParams

pub fn with_throughput_logging(self) -> Self

Enable runner throughput logging.

pub fn with_topology(self, topology: Topology) -> Self

Set the model topology for use during loading. If there is an overlap, the topology type is used over the ISQ type.

pub fn with_tokenizer_json(self, tokenizer_json: impl ToString) -> Self

Path to a discrete tokenizer.json file.

pub fn with_loader_type(self, loader_type: EmbeddingLoaderType) -> Self

Manually set the model loader type. Otherwise, it will attempt to automatically determine the loader type.

pub fn with_dtype(self, dtype: ModelDType) -> Self

Load the model in a certain dtype.

pub fn with_force_cpu(self) -> Self

Force usage of the CPU device. Do not use PagedAttention with this.

pub fn with_token_source(self, token_source: TokenSource) -> Self

Source of the Hugging Face token.

pub fn with_hf_revision(self, revision: impl ToString) -> Self

Set the revision to use for a Hugging Face remote model.

pub fn with_isq(self, isq: IsqType) -> Self

Use ISQ of a certain type. If there is an overlap, the topology type is used over the ISQ type.

pub fn with_logging(self) -> Self

Enable logging.

pub fn with_device_mapping(self, device_mapping: DeviceMapSetting) -> Self

Provide metadata to initialize the device mapper.

pub fn from_uqff(self, path: Vec<PathBuf>) -> Self

👎Deprecated: Use UqffTextModelBuilder to load a UQFF model instead of the generic from_uqff

Path to read a .uqff file from. Other necessary configuration files must be present at this location.

For example, these include:

• residual.safetensors
• tokenizer.json
• config.json
• More depending on the model

pub fn write_uqff(self, path: PathBuf) -> Self

Path to write a .uqff file to and serialize the other necessary files.

The parent (part of the path excluding the filename) will determine where any other files serialized are written to.

For example, these include:

• residual.safetensors
• tokenizer.json
• config.json
• More depending on the model

pub fn from_hf_cache_path(self, hf_cache_path: PathBuf) -> Self

Cache path for Hugging Face models downloaded locally

pub fn with_device(self, device: Device) -> Self

Set the main device to load this model onto. Automatic device mapping will be performed starting with this device.

pub async fn build(self) -> Result<Model>

Trait Implementations

impl Clone for EmbeddingModelBuilder

fn clone(&self) -> EmbeddingModelBuilder

Returns a duplicate of the value.

fn clone_from(&mut self, source: &Self)

Performs copy-assignment from source.

impl From<UqffEmbeddingModelBuilder> for EmbeddingModelBuilder

fn from(value: UqffEmbeddingModelBuilder) -> Self

Converts to this type from the input type.