📐 How to size GPUs for an open-source LLM: the three numbers that matter
Weights are the easy part. KV cache multiplied by concurrency is what actually decides how many GPUs you need — and it's the number most first estimates leave out.
Ask most teams how much GPU memory a model needs and they will multiply parameters by bytes and stop. That answer is right for a demo and wrong for production. At real traffic the attention cache is frequently larger than the model itself. This guide walks through the full calculation so you can reproduce it for any model and workload.
1. Weights
Weights memory ≈ parameters × bytes per parameter, plus roughly 10 % for the runtime. Bytes per parameter depends on precision:
| Precision | Bytes / param | 32B model | 70B model |
|---|---|---|---|
| FP16 / BF16 | 2.0 | ~70 GB | ~155 GB |
| INT8 | ~1.05 | ~37 GB | ~81 GB |
| INT4 (AWQ / GPTQ) | ~0.6 | ~21 GB | ~47 GB |
INT4 is the production default for most chat and retrieval workloads; quality loss is small and measurable, and it halves memory versus INT8.
2. KV cache per request
Every token in every active request keeps its attention keys and values resident. Per token, in FP16:
kv_bytes_per_token = 2 (K and V) × layers × kv_heads × head_dim × 2 bytesFor a 32B model with 64 layers, 8 KV heads and head dim 128 that is about 262 KB per token, so a 4,000-token request holds roughly 1 GB. Grouped-query attention (few KV heads) is why modern models are far cheaper here than older ones with 32 or more KV heads.
3. Concurrency
Concurrency is requests per second multiplied by how long each request stays in flight. A request with 400 output tokens at 30 tokens/second lives for about 13 seconds; at 10 requests per second that is ~130 in flight, each holding its own KV cache.
Putting it together
Total ≈ weights + (KV per request × concurrency) + 20 % headroom. Then fit it onto real GPUs: one 80 GB card, two 48 GB cards with tensor parallelism, or replicas behind a load balancer. Aim for 75–85 % utilisation at peak; above 90 % you will see latency spikes as batches wait for memory.
What moves the number
- Prompt length. Tighter retrieval (fewer, better chunks) cuts input tokens and is often the cheapest optimisation available.
- Prefix caching. Shared system prompts can be cached across requests in vLLM and SGLang, removing that portion of KV cost.
- Quantised KV cache. FP8 KV halves the cache with minor quality impact on recent GPUs.
- Peak vs average. Size for sustained peak. Autoscaling helps only if your cold-start time is acceptable.
The assessment on this site runs exactly this calculation for the models it recommends and shows the breakdown, the range and the GPU options side by side.