Function token_bytes_from_tokenizer_json
pub fn token_bytes_from_tokenizer_json(
tokenizer_json: &Value,
) -> Result<Vec<Vec<u8>>, Error>Expand description
Parse HF tokenizer.json file and return bytes for every token