Function llg_free_tokenizer
#[unsafe(no_mangle)]pub unsafe extern "C" fn llg_free_tokenizer(tok: *mut LlgTokenizer)Expand description
Free the tokenizer. Should NOT be called while there are still constraints using it.
ยงSafety
This function should only be called from C code.