Struct VisionModelBuilder

pub struct VisionModelBuilder { /* private fields */ }

Configure a vision model with the various parameters for loading, running, and other inference behaviors.

Implementations

impl VisionModelBuilder

pub fn new(model_id: impl ToString, loader_type: VisionLoaderType) -> Self

A few defaults are applied here:

• Token source is from the cache (.cache/huggingface/token)
• Maximum number of sequences running is 32

pub fn with_prompt_batchsize(self, prompt_batchsize: NonZeroUsize) -> Self

Set the prompt batchsize to use for inference.

pub fn with_topology(self, topology: Topology) -> Self

Set the model topology for use during loading. If there is an overlap, the topology type is used over the ISQ type.

pub fn with_chat_template(self, chat_template: impl ToString) -> Self

Literal Jinja chat template OR Path (ending in .json) to one.

pub fn with_tokenizer_json(self, tokenizer_json: impl ToString) -> Self

Path to a discrete tokenizer.json file.

pub fn with_dtype(self, dtype: ModelDType) -> Self

Load the model in a certain dtype.

pub fn with_force_cpu(self) -> Self

Force usage of the CPU device. Do not use PagedAttention with this.

pub fn with_token_source(self, token_source: TokenSource) -> Self

Source of the Hugging Face token.

pub fn with_hf_revision(self, revision: impl ToString) -> Self

Set the revision to use for a Hugging Face remote model.

pub fn with_isq(self, isq: IsqType) -> Self

Use ISQ of a certain type. If there is an overlap, the topology type is used over the ISQ type.

pub fn with_calibration_file(self, path: PathBuf) -> Self

Utilise this calibration_file file during ISQ

pub fn with_max_num_seqs(self, max_num_seqs: usize) -> Self

Set the maximum number of sequences which can be run at once.

pub fn with_logging(self) -> Self

Enable logging.

pub fn with_device_mapping(self, device_mapping: DeviceMapMetadata) -> Self

Provide metadata to initialize the device mapper. Generally, it is more programmatic and easier to use the Topology, see Self::with_topology.

pub fn from_uqff(self, path: PathBuf) -> Self

Path to read a UQFF file from.

pub fn from_max_edge(self, max_edge: u32) -> Self

Automatically resize and pad images to this maximum edge length. Aspect ratio is preserved. This is only supported on the Qwen2-VL and Idefics 2 models. Others handle this internally.

pub fn write_uqff(self, path: PathBuf) -> Self

Path to write a UQFF file to.

The parent (part of the path excluding the filename) will determine where any other files generated are written to. These can be used to load UQFF models standalone, and may include:

• residual.safetensors
• tokenizer.json
• config.json
• And others

pub async fn build(self) -> Result<Model>