⚙️ vLLM, SGLang, TensorRT-LLM or llama.cpp: choosing an inference server
The model is half the decision. The server decides your throughput, your latency profile and how much of your GPU you actually use.
Four open inference stacks cover most production deployments. They differ less in what they can run than in what they optimise for.
vLLM
The default choice for multi-user serving on NVIDIA and AMD GPUs. Continuous batching, paged attention, prefix caching, tensor and pipeline parallelism, an OpenAI-compatible API and broad model coverage. If you have no strong reason to pick something else, pick vLLM.
SGLang
Similar scope to vLLM with particularly strong prefix-cache reuse (RadixAttention) and structured-output support. Often ahead on throughput for agentic or multi-turn workloads where prompts share long prefixes.
TensorRT-LLM
NVIDIA's compiled path. Highest raw throughput and lowest latency on NVIDIA hardware, at the cost of a build step per model and GPU type and a narrower model list. Worth it at scale when you have settled on a model.
llama.cpp / Ollama
CPU and consumer-GPU friendly, GGUF quantisation, trivial to run. Ideal for edge devices, laptops and small internal tools; not designed for high-concurrency serving.
Decision rule of thumb
| Situation | Start with |
|---|---|
| Data-centre GPUs, many users | vLLM |
| Agents / long shared prefixes / JSON schemas | SGLang |
| Fixed model, maximum NVIDIA performance | TensorRT-LLM |
| Edge, offline, single user | llama.cpp |
The Updates feed on this site tracks releases of all four so you can see when a new version changes the picture.