Function llg_tokenize_bytes 

#[unsafe(no_mangle)]
pub unsafe extern "C" fn llg_tokenize_bytes(
    tok: &LlgTokenizer,
    bytes: *const u8,
    bytes_len: usize,
    output_tokens: *mut u32,
    output_tokens_len: usize,
) -> usize

Tokenize the given bytes and return the tokens. Always returns the number of tokens that would be written to output_tokens if output_tokens_len was large enough.

Safety

This function should only be called from C code.