Troubleshooting
Before debugging setup issues, run mistralrs doctor. It reports detected hardware, compiled accelerator features, and Hugging Face connectivity.
For unlisted issues, file an issue on GitHub with a reproducer.
Installation and build
Section titled “Installation and build”mistralrs: command not found after install
Section titled “mistralrs: command not found after install”Installer-managed binaries are exposed at ~/.local/bin/mistralrs, backed by ~/.mistralrs/mistralrs. Open a new shell or run source "$HOME/.mistralrs/env". Manual Cargo installs use ~/.cargo/bin/mistralrs and may require source "$HOME/.cargo/env".
Build fails with flash-attn feature enabled
Section titled “Build fails with flash-attn feature enabled”Flash attention requires compute capability 8.0+ (see hardware support). On older GPUs, drop flash-attn and rebuild:
cuda nccl cudnnon Linux with NCCL installed.cuda cudnnotherwise.
mistralrs login rejects the token
Section titled “mistralrs login rejects the token”The token must start with hf_. The validation happens in mistralrs login before saving.
Model loading
Section titled “Model loading”Gated repository (Gemma, LLaMA, FLUX.1-dev, etc.)
Section titled “Gated repository (Gemma, LLaMA, FLUX.1-dev, etc.)”Accept the license on the model’s Hugging Face page, then save a token with mistralrs login. The token is stored at ~/.cache/huggingface/token (or $HF_HOME/token).
Out of memory on load
Section titled “Out of memory on load”Add --quant 4. If still too large, try --quant 2 or split across GPUs with -n "0:N1;1:N2;...". See quantize a model.
Runtime
Section titled “Runtime”Generation slower than expected
Section titled “Generation slower than expected”Verify accelerator features are compiled in with mistralrs doctor. If cuda is missing, the binary was built without GPU support.
For CUDA decode throughput, also check whether paged attention is active. FlashInfer (a CUDA attention backend) paged decode and CUDA graphs are enabled by default for compatible CUDA paged decode paths.
CUDA graphs do not appear to help
Section titled “CUDA graphs do not appear to help”CUDA graphs apply to supported single-token decode steps only. They do not speed up prompt prefill. The first time a graph shape is seen, mistral.rs pays warmup and capture overhead; steady-state decode is the part that can improve.
If graph capture or replay fails, mistral.rs logs a warning and disables CUDA graphs for that loaded pipeline. Set MISTRALRS_CUDA_GRAPHS=0 to compare with the normal CUDA path. See CUDA graphs.
Response cut off
Section titled “Response cut off”max_tokens is most likely too low. Check finish_reason:
length- token limit reached.stop- generation ended naturally (EOS token) or a configured stop token/string matched.
HTTP server
Section titled “HTTP server”Connection refused hitting localhost
Section titled “Connection refused hitting localhost”Check the Server listening on http://... line in the server output to confirm host and port.
CORS errors in a browser
Section titled “CORS errors in a browser”The default allows any origin. Custom CORS configuration is only available programmatically through MistralRsServerRouterBuilder.
413 Payload Too Large
Section titled “413 Payload Too Large”The default body limit is 50 MB and is not configurable via the CLI. Configure programmatically through MistralRsServerRouterBuilder.
UI does not load at /ui
Section titled “UI does not load at /ui”The UI is on by default. Check that --no-ui was not passed at startup, and that no reverse proxy is rewriting /ui.
Sessions
Section titled “Sessions”Sessions disappear between requests
Section titled “Sessions disappear between requests”The session expired (30-minute idle TTL) or was evicted (128-session cap, LRU). Long-lived sessions need explicit export/import via /v1/sessions/{id}. See persist sessions.
Python SDK
Section titled “Python SDK”from mistralrs import Runner fails with ImportError
Section titled “from mistralrs import Runner fails with ImportError”The wrong wheel was installed. pip install mistralrs gives the CPU (Linux/Windows) or Metal (macOS) wheel; for NVIDIA, install a CUDA wheel from the release with --find-links and the +cudaNNN.smNN matching your driver and GPU. See Python SDK getting started.
Rust SDK
Section titled “Rust SDK”ModelBuilder::build() requires a tokio runtime
Section titled “ModelBuilder::build() requires a tokio runtime”The SDK requires a running tokio runtime. Use #[tokio::main] or create a runtime with tokio::runtime::Runtime::new().