Crate toktrie

Modules

bytes

recognizer

rng

Structs

ApproximateTokEnv

Branch

InferenceCapabilities

Defines what is allowed in Branch

SimpleVob

SimpleVobIter

Splice

Describes what to do after sampling.

StepArg

TokEnvWithTrie

TokRxInfo

TokTrie

TrieNode

Constants

INVALID_TOKEN

Traits

Recognizer

TokenizerEnv

Functions

parse_numeric_token

Parse a special token of the form \xFF [ 1 2 3 4 ] The initial \xFF is not included in the input. Returns the number of bytes consumed and the token id.

Type Aliases

StepResult

TokEnv

TokenId