Struct Matcher

pub struct Matcher(/* private fields */);

This is meant to be used in server-side scenarios. The Constraint interface is more for usage in Python Guidance.

Implementations

impl Matcher

pub fn new(parser: Result<TokenParser, Error>) -> Matcher

pub fn deep_clone(&self) -> Matcher

pub fn consume_tokens(&mut self, tokens: &[u32]) -> Result<(), Error>

Advance the parser by one token. Also checks if the parser should stop after consuming the tokens and puts the parser in stop state if necessary.

pub fn consume_token(&mut self, token: u32) -> Result<(), Error>

pub fn rollback(&mut self, num_tokens: usize) -> Result<(), Error>

pub fn reset(&mut self) -> Result<(), Error>

pub fn compute_mask(&mut self) -> Result<SimpleVob, Error>

Compute which tokens can be consumed in the current state.

pub fn compute_mask_or_eos(&mut self) -> Result<SimpleVob, Error>

Compute which tokens can be consumed in the current state. Returns a mask with just the EOS token if the parser is stopped. May still fail if the parser is in an error state.

pub fn is_accepting(&mut self) -> Result<bool, Error>

Can the grammar be finished in the current state? In other words, would the current token mask allow EOS token?

pub fn is_stopped(&self) -> bool

pub fn stop_reason(&self) -> StopReason

pub fn compute_ff_tokens(&mut self) -> Vec<u32>

This will always return [] for non-canonical tokenizers.

pub fn consume_ff_tokens(&mut self) -> Vec<u32>

pub fn compute_ff_bytes(&mut self) -> Vec<u8>

Return any bytes that are forced by the current parser state. This also works for non-canonical tokenizers.

pub fn try_consume_tokens(&mut self, tokens: &[u32]) -> Result<usize, Error>

Tries to advance the parser by consuming the given tokens. Returns the number of tokens consumed. Also checks if the parser should stop after consuming the tokens and puts the parser in stop state if necessary.

pub fn validate_tokens(&mut self, tokens: &[u32]) -> Result<usize, Error>

pub fn is_error(&self) -> bool

pub fn get_error(&self) -> Option<String>

pub fn grammar_warnings(&mut self) -> Vec<String>

pub fn tok_env(&self) -> Result<Arc<dyn TokenizerEnv + Sync>, Error>

pub fn last_step_stats(&self) -> Result<&ParserStats, Error>

Trait Implementations

impl Clone for Matcher

fn clone(&self) -> Matcher

Returns a copy of the value.

fn clone_from(&mut self, source: &Self)

Performs copy-assignment from source.