Function llg_tokenize_bytes

pub extern "C" fn llg_tokenize_bytes(
    tok: &LlgTokenizer,
    bytes: *const u8,
    bytes_len: usize,
    output_tokens: *mut u32,
    output_tokens_len: usize,
) -> usize

Tokenize the given bytes and return the tokens. Always returns the number of tokens that would be written to output_tokens if output_tokens_len was large enough.