Quantization stores and computes model weights in fewer bits — 8, 4, even 2 — instead of 16 or 32. A 70B model in fp16 needs 140GB; in 4-bit, 35GB. Modern quantization loses little quality on most tasks and makes local deployment of frontier-scale models tractable.
- 01Explain what quantization compresses.
- 02State typical quality/size tradeoffs.
- 03Distinguish weight-only from activation quantization.
The compression
Store each weight as a low-bit integer with a shared scale, instead of a 16-bit float. GGUF (llama.cpp), AWQ, and GPTQ are the three dominant formats. 8-bit is nearly lossless; 4-bit costs a few points on hard benchmarks; 2-bit degrades noticeably.
Weight vs. activation
Weight-only quantization shrinks storage and reduces memory bandwidth (the bottleneck for most inference). Activation quantization additionally speeds up computation on hardware that supports low-precision math. Most local runtimes quantize weights only.
- Quantization trades a little quality for a lot of size.
- 4-bit is the standard for local deployment as of 2026.