Skip to content

Distributed inference

When a model doesn’t fit on one GPU, or you want it to run faster on several, mistral.rs can spread it out in three ways:

Mode How it splits the model Use it when
NCCL tensor parallelism Every layer is split across all GPUs, and they work on each token together. You have similar NVIDIA GPUs, on one machine or several. This is the default, high-throughput path.
Layer mapping Each GPU holds a different range of whole layers, and each token passes from one GPU to the next. Your GPUs differ in size or speed, the model can’t be split evenly across your GPU count, or you don’t have NCCL.
Ring backend Like tensor parallelism, but between machines over plain TCP instead of NCCL. You want to split a model across machines that can’t use NCCL, such as Macs.

With a CUDA build and more than one visible GPU, mistral.rs picks a mode for you:

  • If the binary includes NCCL, it uses tensor parallelism. Linux CUDA prebuilt binaries include NCCL.
  • Otherwise, it uses layer mapping and gives each GPU a share of the layers that fits its free memory.

The startup log says which layout it chose. Multi-node NCCL and the ring backend are never picked automatically; you turn them on with the environment variables below.

The Linux CUDA prebuilt binaries from the installer already include NCCL. Installer source builds include it when NCCL is installed on the machine; pass --no-nccl to leave it out.

To build by hand with NCCL:

Terminal window
cargo install mistralrs-cli --features "cuda nccl flash-attn cudnn"

Drop nccl from the list if NCCL isn’t installed.

Set CUDA_VISIBLE_DEVICES to choose which GPUs mistral.rs can see:

Terminal window
CUDA_VISIBLE_DEVICES=0,1 mistralrs serve -m Qwen/Qwen3-32B --quant 4

mistral.rs renumbers the visible GPUs from 0. With CUDA_VISIBLE_DEVICES=2,3, refer to them as 0 and 1 in --device-layers.

Tensor parallelism needs the model’s attention heads to split evenly across the GPUs. The KV heads must also split evenly, or, when there are fewer KV heads than GPUs, the GPU count must be a multiple of the KV head count. If that doesn’t work for all visible GPUs, mistral.rs uses the largest GPU count that does and logs the size it chose. For example, Qwen3-32B has 64 attention heads and 8 KV heads, so on 3 GPUs it runs tensor-parallel on 2, and on 6 GPUs it runs on 4. If no count works, the model loads without tensor parallelism. Use CUDA_VISIBLE_DEVICES to control exactly which GPUs take part.

By default, layer mapping gives each GPU a contiguous range of layers, in GPU order, sized to that GPU’s free memory. It budgets weights, working memory, and the KV cache together; see memory budget.

It doesn’t account for GPU speed or how the GPUs are connected (PCIe or NVLink, same or different CPU socket). When those differ, set the layer counts yourself, for example putting fewer layers on a smaller, slower, or busier GPU:

-n (or --device-layers) takes GPU:LAYERS pairs separated by semicolons:

Terminal window
# Qwen3-32B has 64 layers: even split
mistralrs serve -n "0:32;1:32" -m Qwen/Qwen3-32B --quant 4
# Uneven split
mistralrs serve -n "0:44;1:20" -m Qwen/Qwen3-32B --quant 4

On a binary with NCCL, tensor parallelism takes over and --device-layers is ignored. Set MISTRALRS_NO_NCCL=1 to use layer mapping instead:

Terminal window
MISTRALRS_NO_NCCL=1 mistralrs serve -n "0:44;1:20" -m Qwen/Qwen3-32B --quant 4

Data moves between GPUs only where one GPU’s layers end and the next GPU’s begin. It goes straight from GPU to GPU when the driver allows it; otherwise it’s copied through CPU memory, which is slower, and the startup log shows a warning.

If the model doesn’t fit across your GPUs, automatic placement puts the remaining layers on the CPU, which is much slower and turns paged attention off. If you’ve set --kv-cache or --gpu-memory-utilization, loading fails instead.

For per-layer or per-tensor placement, see the topology guide.

Multi-node NCCL runs tensor parallelism across several machines, each with one or more NVIDIA GPUs. Every node needs:

  • The same CUDA and NCCL build, model, dtype, quantization, and flags.
  • The same number of visible GPUs. The total GPU count (GPUs per node times nodes) must split the model’s heads evenly, as described above. Unlike on a single machine, a count that doesn’t work fails at startup.
  • No MISTRALRS_NO_NCCL or RING_CONFIG set.

Configure each node with these environment variables:

Variable Set on Value
MISTRALRS_MN_GLOBAL_WORLD_SIZE Every node Total GPUs across all nodes. Setting it turns on multi-node mode.
MISTRALRS_MN_LOCAL_WORLD_SIZE Every node GPUs on this node.
MISTRALRS_MN_HEAD_NUM_WORKERS Head node Number of worker nodes.
MISTRALRS_MN_HEAD_PORT Head node Port the workers connect to.
MISTRALRS_MN_WORKER_SERVER_ADDR Worker nodes Head node’s address and port.
MISTRALRS_MN_WORKER_ID Worker nodes This worker’s number, starting at 0.

Two nodes with four GPUs each:

Head node:

Terminal window
CUDA_VISIBLE_DEVICES=0,1,2,3 \
MISTRALRS_MN_GLOBAL_WORLD_SIZE=8 \
MISTRALRS_MN_LOCAL_WORLD_SIZE=4 \
MISTRALRS_MN_HEAD_NUM_WORKERS=1 \
MISTRALRS_MN_HEAD_PORT=9000 \
mistralrs serve -m Qwen/Qwen3-32B --quant 4

Worker node:

Terminal window
CUDA_VISIBLE_DEVICES=0,1,2,3 \
MISTRALRS_MN_GLOBAL_WORLD_SIZE=8 \
MISTRALRS_MN_LOCAL_WORLD_SIZE=4 \
MISTRALRS_MN_WORKER_SERVER_ADDR=10.0.0.1:9000 \
MISTRALRS_MN_WORKER_ID=0 \
mistralrs serve -m Qwen/Qwen3-32B --quant 4

Send requests to the head node. The head’s port (9000 here) must be reachable from every worker.

The ring backend connects machines in a loop over TCP, with each machine running one part of the model. Build with the ring feature on every machine:

Terminal window
# NVIDIA GPUs
cargo install mistralrs-cli --features "cuda flash-attn ring"
# Apple Silicon
cargo install mistralrs-cli --features "metal ring"

If the binary also includes nccl, set MISTRALRS_NO_NCCL=1 when launching so the ring backend is used.

Give each machine its own JSON config and point RING_CONFIG at it. For two machines at 10.0.0.1 and 10.0.0.2:

{
"master_ip": "10.0.0.1",
"master_port": 9000,
"port": 9001,
"right_ip": "10.0.0.2",
"right_port": 9002,
"rank": 0,
"world_size": 2
}
Terminal window
RING_CONFIG=ring.json mistralrs serve -m Qwen/Qwen3-32B --quant 4
Field Meaning
rank This machine’s position in the ring, from 0 to world_size - 1. Rank 0 is the master: send requests to it.
world_size Number of machines. Must be a power of 2 (2, 4, 8, and so on).
master_ip, master_port Where the other ranks reach rank 0. Required on every rank except 0.
port Port this machine listens on for the previous rank.
right_ip, right_port Address and port of the next rank. The last rank points back to rank 0.

Each rank waits only 10 seconds to reach the next one, so start all machines at about the same time.