Observability

mistralrs serve writes operational logs through the normal tracing subscriber, so access logs appear in the same stdout/stderr stream as startup logs. mistralrs from-config uses the same server path and the same defaults.

Access logs

Access logs are enabled by default. For each non-housekeeping request, the server logs:

• request started: request id, method, matched route, raw path, resolved model id, and request content length when present.
• request completed: request id, method, matched route, resolved model id, status, and duration in milliseconds rounded to three decimal places.

Housekeeping endpoints and CORS preflight requests are skipped by default. Housekeeping endpoints include /, /health, /metrics, /docs, /api-doc/openapi.json, and /ui. Use --access-log-health if you want probes and docs/UI requests in the access log too.

Useful flags:

Flag	Default	Effect
--disable-access-log	false	Turns off info-level HTTP access logs.
--access-log-format text	text	Uses structured tracing fields for access logs.
--access-log-format json	text	Logs each access-log message as a JSON object string.
--access-log-health	false	Includes housekeeping endpoints in access logs.

The global -v, -vv, and RUST_LOG controls still decide which tracing levels are emitted. --disable-access-log only disables the info-level HTTP access log events.

Caution

-l, --log <path> is not an access-log file. It records full model requests and responses. Only use it where payloads can be stored safely.

Request ids

The server honors an incoming x-request-id header when present, otherwise it generates one. The value is added to access logs and echoed as the x-request-id response header.

The default CORS configuration allows browser clients to send x-request-id and exposes the response header.

Use --disable-request-id-header to stop adding the response header. The request id is still generated internally for access logs.

Prometheus metrics

GET /metrics exposes Prometheus text format by default. Metrics come from two layers: the HTTP middleware (http_*) and the inference engine (mistralrs_*). The engine metrics are unlabeled because the engine is process-wide; use one server process per model for per-model dashboards, or add labels in your scrape/relabelling layer.

HTTP metrics

The HTTP middleware records:

Metric	Type	Labels	Meaning
http_requests_total	counter	method, path, model, status	Completed request count.
http_request_duration_seconds	histogram	method, path, model, status	End-to-end request latency.
http_requests_in_flight	gauge	method, path, model	Requests currently running.
http_request_body_bytes	histogram	method, path, model	Request body bytes when the body size is known.

Streaming latency

For streaming (text/event-stream) responses, the middleware also records per-stream latency:

Metric	Type	Labels	Meaning
mistralrs_time_to_first_token_seconds	histogram	method, path, model, status	Time from request start to the first engine token step. Includes queueing and prefill.
mistralrs_inter_token_latency_seconds	histogram	method, path, model, status	Gaps between consecutive engine token steps. One sample per step after the first.

Timing is measured at the engine’s token steps, not at SSE frames, so keep-alive pings (axum comment frames and Anthropic event: ping) and protocol events ([DONE], message_start/message_stop, and similar bookkeeping frames) never register as samples. Long decode stalls show up as high ITL quantiles, which a per-request average would hide. Both histograms use the same bucket boundaries as vLLM’s exporter; the existing http_request_* histograms keep their own bucket lists.

Engine metrics

Metric	Type	Labels	Meaning
mistralrs_tokens_processed_total	counter	-	Tokens processed (prefill plus decode). Always equals the sum of the two phase counters below.
mistralrs_prefill_tokens_processed_total	counter	-	Prompt (prefill) tokens processed.
mistralrs_decode_tokens_processed_total	counter	-	Generated (decode) tokens processed. With speculative decoding only verified tokens count.
mistralrs_prefix_cache_lookups_total	counter	-	New sequences that requested a prefix cache lookup.
mistralrs_prefix_cache_hits_total	counter	-	New sequences that found a cached KV prefix.
mistralrs_prefix_cache_evictions_total	counter	-	Prefix cache entries, logical recurrent owners, or paged KV hash keys evicted from cache.
mistralrs_encoder_cache_hits_total	counter	-	Multimodal encoder cache hits (multimodal models only).
mistralrs_encoder_cache_misses_total	counter	-	Multimodal encoder cache misses (multimodal models only).
mistralrs_encoder_cache_intra_batch_deduplications_total	counter	-	Repeated media items coalesced within one encoder batch.
mistralrs_encoder_cache_evictions_total	counter	reason	Encoder outputs removed because of entry, configured logical-byte, or shape constraints.
mistralrs_encoder_cache_insert_rejections_total	counter	reason	Encoder outputs rejected because one entry exceeds the configured byte budget or compact storage allocation failed.
mistralrs_encoder_cache_resident_entries	gauge	-	Encoder outputs resident across all encoder caches in the process.
mistralrs_encoder_cache_resident_logical_bytes	gauge	-	Logical tensor payload bytes resident across all encoder caches in the process. Byte-budgeted entries use compact storage.
mistralrs_sequences_running	gauge	-	Sequences currently running.
mistralrs_sequences_waiting	gauge	-	Sequences in the waiting queue.
mistralrs_sequences_capacity	gauge	-	Maximum sequences the engine schedules concurrently (set at engine startup).
mistralrs_sequences_completed_total	counter	reason	Sequences that reached a terminal state. reason is one of stop, length, canceled, generated_image, generated_speech, tool_calls, error.
mistralrs_kv_cache_blocks_used	gauge	-	KV cache blocks currently in use (paged attention).
mistralrs_kv_cache_blocks_total	gauge	-	Total KV cache blocks (paged attention).
mistralrs_kv_cache_blocks_active	gauge	-	Physical KV cache blocks referenced by running sequences.
mistralrs_kv_cache_blocks_reserved	gauge	-	Prompt-tail blocks logically reserved but not yet materialized.
mistralrs_kv_cache_blocks_available	gauge	-	Physical KV blocks available after subtracting prompt-tail reservations.
mistralrs_kv_cache_blocks_prefix_cached	gauge	-	Physical KV blocks that retain at least one prefix-cache key.
mistralrs_kv_cache_blocks_prefix_retained	gauge	-	Physical KV blocks protected by a paired recurrent prefix checkpoint.
mistralrs_paged_prefix_retention_free_blocks	gauge	-	Protected physical KV blocks not referenced by a running sequence.
mistralrs_paged_prefix_retention_entries	gauge	-	Active paired recurrent checkpoint leases. This drops immediately when allocator pressure revokes a lease.
mistralrs_paged_prefix_retention_hashes	gauge	-	Distinct block hashes protected by active paired checkpoint leases.
mistralrs_paged_prefix_retention_pressure_evictions_total	counter	reason	Paired checkpoint leases revoked to preserve allocation progress. reason is allocation_pressure or reset.
mistralrs_paged_recurrent_prefix_owners_used	gauge	-	Logical recurrent prefix owners currently retained.
mistralrs_paged_recurrent_prefix_owners_capacity	gauge	-	Configured logical recurrent prefix owner capacity.
mistralrs_paged_recurrent_prefix_checkpoints_used	gauge	-	Recurrent prefix checkpoints currently retained.
mistralrs_paged_recurrent_prefix_bytes_used	gauge	-	Logical tensor payload bytes retained by recurrent prefix checkpoints.
mistralrs_paged_recurrent_prefix_validation_total	counter	outcome, reason	Prefix validation outcomes, including attention-only hits and recurrent restore misses.
mistralrs_paged_recurrent_prefix_owner_evictions_total	counter	reason	Logical recurrent owners evicted by capacity or after block_pressure revokes their paired blocks.

Speculative decoding (MTP)

These metrics are only populated when a model with a speculative proposer (MTP) is attached. position is the 0-based draft position within a proposal.

Metric	Type	Labels	Meaning
mistralrs_speculative_drafts_total	counter	-	Verified sequences: one per sequence per step that used speculative verification.
mistralrs_speculative_draft_tokens_proposed_total	counter	-	Draft tokens proposed for verification.
mistralrs_speculative_draft_tokens_accepted_total	counter	-	Draft tokens the target model accepted.
mistralrs_speculative_draft_tokens_accepted_per_pos_total	counter	position	Draft tokens accepted at each proposal position. The drop-off across positions shows where proposals start failing.
mistralrs_speculative_staged_drops_total	counter	-	Staged proposals discarded without verification (preemption, batch-shape mismatch, or a step that cannot verify).
mistralrs_paged_preemptions_total	counter	-	Running sequences preempted and returned to the waiting queue (paged attention).

Useful PromQL

Acceptance is a ratio of cumulative counters, so rates and quantiles belong in PromQL rather than extra gauges:

# Fraction of proposed draft tokens that were accepted
rate(mistralrs_speculative_draft_tokens_accepted_total[5m])
  / rate(mistralrs_speculative_draft_tokens_proposed_total[5m])

# Mean tokens accepted per verification step (1.0 = speculation is not helping)
1 + rate(mistralrs_speculative_draft_tokens_accepted_total[5m])
  / rate(mistralrs_speculative_drafts_total[5m])

# P99 time to first token, all models
histogram_quantile(0.99,
  sum(rate(mistralrs_time_to_first_token_seconds_bucket[5m])) by (le))

# P95 inter-token latency for one route/model
histogram_quantile(0.95,
  sum(rate(mistralrs_inter_token_latency_seconds_bucket{path="/v1/chat/completions"}[5m])) by (le))

# KV cache pressure (sustained 1.0 with a growing waiting queue means preemptions are likely)
mistralrs_kv_cache_blocks_used / mistralrs_kv_cache_blocks_total

# Sequence capacity utilization (sustained 1.0 with a growing waiting queue means requests are queueing)
mistralrs_sequences_running / mistralrs_sequences_capacity

If you want gauge-like ergonomics for the acceptance rate, use a recording rule instead of a server-side percentile gauge:

groups:
  - name: mistralrs
    rules:
      - record: mistralrs:mtp_accept_rate:rate5m
        expr: |
          rate(mistralrs_speculative_draft_tokens_accepted_total[5m])
            / rate(mistralrs_speculative_draft_tokens_proposed_total[5m])

The path label is the matched route pattern, such as /v1/responses/{response_id}, not the raw URI. The model label is the resolved model id for inference requests, defaults to the server default model when the request omits model, uses explicit model_id values for model-management requests, uses unknown when the request body cannot be read or parsed as JSON, and uses none for routes that do not target a model. Unmatched requests are labeled <unmatched>. Housekeeping endpoints are excluded from HTTP metrics to keep scrape and probe traffic out of request dashboards.

Use --disable-metrics to avoid installing the Prometheus recorder. When metrics are disabled or not initialized, GET /metrics returns 503.

TOML

The same controls are available under [server]:

[server]
host = "0.0.0.0"
port = 1234

disable_access_log = false
access_log_format = "text" # "text" or "json"
access_log_health = false
disable_request_id_header = false
disable_metrics = false

For endpoint semantics, see the HTTP API reference. For deployment checks, see the production checklist.