Function llg_new_tokenizer

pub unsafe extern "C" fn llg_new_tokenizer(
    tok_init: &LlgTokenizerInit,
    error_string: *mut i8,
    error_string_len: usize,
) -> *mut LlgTokenizer

Construct a new tokenizer from the given TokenizerInit

Safety

This function should only be called from C code.