Summary
An eval is a test suite for a prompt: a fixed set of inputs, expected outputs, and a scoring rule. Without evals, you cannot know whether a prompt change is an improvement or a regression. The single discipline that separates working systems from working demos.
Objectives
- 01Design an eval for a prompt.
- 02Choose between rule-based, LLM-as-judge, and human scoring.
- 03Track eval scores over time and across prompt versions.
The Lesson
The components
Test set (50-500 inputs), expected outputs or scoring rubric, scoring function, threshold for pass. Run evals before every prompt change; refuse merges that regress.
Scoring choices
Rule-based (regex, JSON schema match): fast, precise, limited to structured output. LLM-as-judge: flexible, cheap, needs validation against humans. Human scoring: gold standard, slow, expensive. Most production systems combine all three.
Key Ideas
- Evals are the difference between engineering and folklore.
- Rule-based + LLM-judge + occasional human is the practical stack.