Summary
The quality-loss story is mostly honest: 8-bit is nearly free, 4-bit loses ~1-3 points on hard benchmarks, 2-bit degrades noticeably. But 'quality loss' averages hide task-specific behavior. Code and math often degrade more than casual chat; long-context often degrades more than short.
Objectives
- 01State typical benchmark losses at each precision.
- 02Recognize task-specific degradation patterns.
- 03Explain why you should measure on your task.
The Lesson
Benchmarks vs. reality
Aggregate MMLU or HellaSwag drops of 1-2 points at 4-bit hide larger losses on GSM8K (math), HumanEval (code), or long-context QA. If your task is math-heavy or code-heavy, run your evaluation on both fp16 and 4-bit before committing to quantization.
Measure your task
Aggregate quality is a useful prior. Task quality is the number that matters. Every serious deployment measures both — the delta on your task is what determines whether quantization is free or expensive.
Key Ideas
- Averages hide task-specific quantization loss.
- Measure on your task; do not trust the leaderboard.