Function llg_free_tokenizer

pub unsafe extern "C" fn llg_free_tokenizer(tok: *mut LlgTokenizer)

Free the tokenizer. Should NOT be called while there are still constraints using it.

Safety

This function should only be called from C code.