Type Alias LlgTokenizeFn

pub type LlgTokenizeFn = Option<extern "C" fn(_: *const c_void, _: *const u8, _: usize, _: *mut u32, _: usize) -> usize>;

Tokenization function Will not write more than output_tokens_len tokens (which can be 0) Returns the total number of tokens (which can be more than output_tokens_len) This function has to be thread-safe!

Aliased Type

enum LlgTokenizeFn {
    None,
    Some(extern "C" fn(_: *const c_void, _: *const u8, _: usize, _: *mut u32, _: usize) -> usize),
}

Variants

None

No value.

Some(extern "C" fn(_: *const c_void, _: *const u8, _: usize, _: *mut u32, _: usize) -> usize)

Some value of type T.