Function llg_tokenize_bytes_marker

pub extern "C" fn llg_tokenize_bytes_marker(
    tok: &LlgTokenizer,
    bytes: *const u8,
    bytes_len: usize,
    output_tokens: *mut u32,
    output_tokens_len: usize,
) -> usize

Tokenize the given bytes and return the tokens. Special tokens will be tokenized, if they follow 0xFF byte prefix. Always returns the number of tokens that would be written to output_tokens if output_tokens_len was large enough.