🗜️ Quantization in practice: AWQ, GPTQ, GGUF and FP8
Quantized weights are how most open models reach production. Here is what each format is for, what it costs in quality, and how to verify it on your own data.
Quantization stores weights in fewer bits than the model was trained in. Done well it halves or quarters memory with a quality loss you can measure and accept; done carelessly it silently degrades the tasks you care about.
Formats
| Format | Bits | Best for | Servers |
|---|---|---|---|
| AWQ | 4 | GPU serving; good quality at INT4 | vLLM, SGLang, TensorRT-LLM |
| GPTQ | 4 / 8 | GPU serving; widely available | vLLM, SGLang |
| FP8 | 8 | H100/H200 and newer; near-lossless | vLLM, SGLang, TensorRT-LLM |
| GGUF | 2–8 | CPU / consumer GPU; llama.cpp | llama.cpp, Ollama |
| bitsandbytes | 4 / 8 | Quick experiments in Transformers | Transformers |
How much quality do you lose?
For chat, retrieval and summarisation, INT4 AWQ/GPTQ typically costs a point or two on general benchmarks. Losses are larger on precise numeric reasoning and on very small models (under 7B), where each bit matters more. FP8 is close to lossless where hardware supports it.
Verify on your own data
- Build a 100–300 example evaluation set from real requests with reference answers.
- Run the FP16 and quantized variants on the same prompts through the same server.
- Compare with your task metric (exact match, rubric score, human preference), not just perplexity.
- Ship the quantized variant only when the delta is inside your tolerance.
The catalog lists which quantized variants exist on the Hub for each model, so you can tell at a glance whether a production-ready AWQ or FP8 build is available.