Expand description
Provides the default values used for the mistral.rs instance for server. These defaults can be used for CLI argument fallbacks, config loading, or general initialization.
Constantsยง
- CHAT_
TEMPLATE - CPU
- DEVICE
- ENABLE_
SEARCH - INTERACTIVE_
MODE - IN_
SITU_ QUANT - JINJA_
EXPLICIT - LOG
- MAX_
SEQS - MODEL
- NO_
KV_ CACHE - NUM_
DEVICE_ LAYERS - PAGED_
ATTN - PAGED_
ATTN_ BLOCK_ SIZE - PAGED_
ATTN_ CPU - PAGED_
ATTN_ CUDA - PAGED_
ATTN_ GPU_ MEM - PAGED_
ATTN_ GPU_ MEM_ USAGE - PAGED_
ATTN_ METAL - PAGED_
CACHE_ TYPE - PAGED_
CTXT_ LEN - PREFIX_
CACHE_ N - PROMPT_
CHUNKSIZE - SEARCH_
BERT_ MODEL - SEARCH_
CALLBACK - SEED
- TOKEN_
SOURCE - TRUNCATE_
SEQUENCE