Summary
Two lesser-used inference knobs. Repetition penalty discourages tokens that recently appeared. Beam search maintains multiple candidate sequences and picks the best overall — dominant in translation, largely absent from chat.
Objectives
- 01State what a repetition penalty does.
- 02Describe beam search in three sentences.
- 03Explain why beam search fell out of favor for open-ended generation.
The Lesson
Repetition penalty
Divide (or subtract from) the logits of tokens that recently appeared, so the model is less likely to loop. Typical value: 1.1-1.3. Useful with small models that repeat; usually unnecessary with frontier models.
Beam search
Maintain k parallel partial sequences ('beams'). At each step, expand each beam, score all candidates, keep the top k. Return the best complete sequence. Dominant in machine translation; produces bland, high-probability output for open-ended chat — one reason it fell out of favor.
Key Ideas
- Repetition penalty combats small-model loops.
- Beam search wins for translation, loses for chat.