Summary
Self-consistency: sample multiple chain-of-thought answers, take the majority vote. Gains another 5-15 percentage points on hard reasoning benchmarks, at the cost of running inference multiple times.
Objectives
- 01Describe the self-consistency algorithm.
- 02State the cost/quality tradeoff.
- 03Explain why it works even when individual chains are unreliable.
The Lesson
The algorithm
Prompt with CoT. Sample k=5-40 completions at moderate temperature (T≈0.7). Extract final answers. Return the most frequent one. Empirically works because incorrect reasoning paths disagree with each other, correct ones agree.
The cost
k=10 samples means 10x the inference cost. Worth it for problems where a wrong answer is expensive. Not worth it for high-volume, low-stakes queries.
Key Ideas
- Sample many CoT answers, vote — beats single CoT.
- Cost is k× inference; use where accuracy justifies it.
References
- Wang et al., 'Self-Consistency Improves Chain of Thought Reasoning' (2022)