Summary
The evaluation lab. A written rubric, human scoring, inter-rater agreement, and an LLM-as-judge validated against the humans. The discipline that lets you improve a system rather than argue about it.
Objectives
- 01Author a rubric with 5+ criteria and defined levels.
- 02Score outputs with at least two independent raters.
- 03Compute inter-rater agreement (Cohen's kappa).
- 04Validate an LLM-as-judge against the human baseline.
Key Ideas
- A rubric is not real until two people score with it and agree.
- LLM-as-judge is a shortcut; the calibration to humans is the work.
Lab
Choose one AI task from a system you or a peer built. Write a rubric with 5+ criteria, each with anchored levels (e.g., 1-4 with descriptions). Sample 50 outputs. Recruit at least one other scholar as a second rater. Both raters score independently. Compute Cohen's kappa on the scores. Then implement an LLM-as-judge with the same rubric. Report the LLM's agreement with each human rater. Ship the rubric, the scored data, and a memo on where the LLM judge is trustworthy and where it is not.
Deliverables
- — Written rubric with anchored levels.
- — 50 outputs scored by two humans.
- — Inter-rater agreement statistics.
- — LLM-as-judge with validation report against humans.
Rubric
- — Rubric quality — criteria are specific and levels are distinguishable.
- — Human agreement — kappa reported honestly, disagreements analyzed.
- — LLM validation — the judge is used only where it agrees with humans.
- — Documentation — a stranger could reproduce the evaluation.