Function llg_stringify_tokens
pub extern "C" fn llg_stringify_tokens(
tok: &LlgTokenizer,
tokens: *const u32,
n_tokens: usize,
output: *mut i8,
output_len: usize,
) -> usize
Expand description
Return a string representation of the tokens, useful for debugging. The output is null-terminated. Returns the number of bytes that would be written to output if output_len was large enough.