Expand description

Provides the default values used for the mistral.rs instance for server. These defaults can be used for CLI argument fallbacks, config loading, or general initialization.

Constantsยง

CHAT_TEMPLATE
CPU
DEVICE
ENABLE_SEARCH
INTERACTIVE_MODE
IN_SITU_QUANT
JINJA_EXPLICIT
LOG
MAX_SEQS
MODEL
NO_KV_CACHE
NO_PAGED_ATTN
NUM_DEVICE_LAYERS
PAGED_ATTN
PAGED_ATTN_BLOCK_SIZE
PAGED_ATTN_GPU_MEM
PAGED_ATTN_GPU_MEM_USAGE
PAGED_CTXT_LEN
PREFIX_CACHE_N
PROMPT_CHUNKSIZE
SEARCH_BERT_MODEL
SEARCH_CALLBACK
SEED
TOKEN_SOURCE
TRUNCATE_SEQUENCE