Summary
Agent evaluation is harder than model evaluation. Success is trajectory-dependent, environments are stateful, and 'the same' task rarely runs the same way twice. The discipline that most agent teams skip — and pay for later.
Objectives
- 01Distinguish outcome from process metrics.
- 02Name three agent-specific evaluation approaches.
- 03State why 'end-to-end task completion' is a lagging indicator.
The Lesson
Metrics
Outcome: did the task complete? Process: did the agent take reasonable steps? Cost: how many tokens/tools did it use? Time: how long did it take? Report all four; optimizing only outcome hides regressions on cost.
Approaches
1. Fixed benchmarks (SWE-bench, WebArena) — comparable but limited. 2. Trajectory replay: replay recorded tasks against new agents. 3. LLM-as-judge on trajectories: score each step's reasonableness. Combine all three.
Key Ideas
- Agent evaluation requires trajectory and outcome metrics together.
- Cost and time are first-class metrics, not afterthoughts.