Quantization stores weights in fewer bits. A weight originally stored as a 16-bit float becomes an 8-bit, 4-bit, or 2-bit integer with a shared scale factor. The tradeoff is quality for size. Below 4 bits, quality drops noticeably; above 4 bits, differences are small on most tasks.
- 01State the size ratio at 8-bit, 4-bit, 2-bit vs. fp16.
- 02Explain what a quantization block is.
- 03State typical quality loss at 4-bit.
The math
Store weights as low-bit integers with a shared scale (and often a zero-point) per block of 32-256 weights. To use: dequantize on the fly. Size: 8-bit = half of fp16, 4-bit = quarter, 2-bit = eighth. Quality: 8-bit nearly indistinguishable, 4-bit ~1-3 points on hard benchmarks, 2-bit meaningful degradation.
Blocks
Smaller blocks (32-64 weights) preserve more precision at slight size overhead. Larger blocks (128-256) trade quality for compactness. Format defaults are usually reasonable; the tuning is at the frontier of research.
- 4-bit is the sweet spot for local deployment.
- Below 4-bit is research territory; above 8-bit rarely pays off.