mistralrs::llguidance::ffi

Function llg_stringify_tokens

pub extern "C" fn llg_stringify_tokens(
    tok: &LlgTokenizer,
    tokens: *const u32,
    n_tokens: usize,
    output: *mut i8,
    output_len: usize,
) -> usize
Expand description

Return a string representation of the tokens, useful for debugging. The output is null-terminated. Returns the number of bytes that would be written to output if output_len was large enough.