Frequently asked questions
How much GPU memory does an open-source LLM need?
Roughly parameters × bytes per parameter for weights (a 32B model is about 20 GB at INT4), plus KV cache that grows with context length and concurrent requests. At peak traffic the KV cache often exceeds the weights.
What is KV cache and why does it matter for sizing?
The KV cache stores attention keys and values for every token in every active request. Its size scales with layers × context length × concurrency, so long contexts and high requests-per-second are the main drivers of GPU memory.
How do I convert requests per second into concurrent requests?
Multiply peak requests per second by the average request duration. Ten requests per second that each take four seconds means about forty requests in flight, and each needs its own KV cache.
Is INT4 quantization safe for production?
For most chat and retrieval workloads INT4 (AWQ or GPTQ) loses little measurable quality and roughly halves memory versus INT8. Validate on your own evaluation set before committing.