Struct LlgTokenizerInit

#[repr(C)]
pub struct LlgTokenizerInit {
    pub vocab_size: u32,
    pub tok_eos: u32,
    pub token_lens: *const u32,
    pub token_bytes: *const u8,
    pub tokenizer_json: *const i8,
    pub tokenize_assumes_string: bool,
    pub tokenize_fn: Option<extern "C" fn(_: *const c_void, _: *const u8, _: usize, _: *mut u32, _: usize) -> usize>,
    pub use_approximate_greedy_tokenize_fn: bool,
    pub tokenize_user_data: *const c_void,
    pub slices: *const *const i8,
}

Fields

vocab_size: u32

The number of tokens in the vocabulary

tok_eos: u32

The token ID for the end of sentence token For chat mode, set it to end-of-turn token

token_lens: *const u32

An array of the lengths of the token strings (vocab_size elements)

token_bytes: *const u8

A pointer to the token strings The length of this the sum of all token_lens

tokenizer_json: *const i8

Instead of passing token_lens and token_bytes, this can be set to the contents of HF tokenizer.json file.

tokenize_assumes_string: bool

Set to true to enable hack that works around the tokenize_fn only accepting valid UTF-8 strings and possibly adding etc. TODO: the bit not implemented yet

tokenize_fn: Option<extern "C" fn(_: *const c_void, _: *const u8, _: usize, _: *mut u32, _: usize) -> usize>

Tokenization function, see LlgTokenizeFn docs. It should only tokenize the bytes and not add any etc. It should also work on any byte sequence, including invalid UTF-8. If this is not the case, set tokenize_assumes_string to true. Either way, this function has to be thread-safe!

use_approximate_greedy_tokenize_fn: bool

Set to true to not use tokenize_fn and instead tokenize greedily, which is often incorrect and may reduce accuracy.

tokenize_user_data: *const c_void

User data to pass to the tokenize_fn

slices: *const *const i8

Tokenizer partitions for the slicer optimization. This is array of pointers to strings, terminated with NULL (argv style). Pass NULL to use defaults. Pass empty array to disable.