Function llg_clone_tokenizer
#[no_mangle]
pub extern "C" fn llg_clone_tokenizer(
tok: &LlgTokenizer,
) -> *mut LlgTokenizer
Expand description
Clone a tokenizer. This increments a reference count and does a small allocation.
#[no_mangle]
pub extern "C" fn llg_clone_tokenizer(
tok: &LlgTokenizer,
) -> *mut LlgTokenizer
Clone a tokenizer. This increments a reference count and does a small allocation.