Struct Constraint

pub struct Constraint {
    pub parser: TokenParser,
    pub log_json_progress: bool,
    pub temperature: f32,
    /* private fields */
}

Fields

parser: TokenParser

log_json_progress: bool

temperature: f32

Implementations

impl Constraint

pub fn new(parser: TokenParser) -> Constraint

Construct a state machine for a sequence constraint.

pub fn deep_clone(&self) -> Constraint

pub fn process_prompt(&mut self, prompt: Vec<u32>) -> Vec<u32>

You can call this first with the prompt from the user, when not running in chat mode. This will return a new prompt, possibly with some tokens added as per the grammar (and possibly with some tokens removed, for token healing).

pub fn force_tokens(&mut self, tokens: &[u32]) -> Result<(), Error>

This can be called before the first compute_mask() to walk forward the parser with tokens generated in some previous run.

pub fn has_pending_stop(&self) -> bool

pub fn compute_mask(&mut self) -> Result<&Branch<SimpleVob>, Error>

This computes token sampling mask. It typically takes up to a millisecond for a 100k tokenizer. It will return an error when the order of calls is violated. The result will be either:

• a mask of allowed tokens to sample, or
• an unconditional splice result, indicating that the parser wants to append tokens, or
• a stop result, indicating that the parser is done The splice is never returned when ff_tokens are disabled in InferenceCapabilities. After this returns, commit_token() must be called with the sampled token if any.

pub fn step_result(&self) -> &Branch<SimpleVob>

pub fn validate_tokens_raw(&mut self, tokens: &[u32]) -> Result<usize, Error>

pub fn commit_token( &mut self, sampled_token: Option<u32>, ) -> Result<CommitResult, Error>

commit_token() is a top-level method in this file and is called by the LLInterpreter::commit_token().

commit_token() commits the sampled token (if any), and sees if this forces any more tokens on the output (if ff_tokens are enabled in InferenceCapabilities).

It only returns ‘STOP’ if previous compute_mask() already returned ‘STOP’ (in which case there’s little point calling commit_token()).

pub fn flush_progress(&mut self) -> Vec<ParserOutput>

This returns parser outputs to be passed back to the user. You can use that for structured output, or set log_json_progress to true and then use flush_logs() to get a string, from which the user can extract the JSON of the outputs.

pub fn flush_logs(&mut self) -> String

Logs to be sent to the user.

pub fn tok_trie(&self) -> &TokTrie

Trait Implementations

impl Clone for Constraint

fn clone(&self) -> Constraint

Returns a copy of the value.

fn clone_from(&mut self, source: &Self)

Performs copy-assignment from source.