Summary
A defensible comparison of frontier and local models on one task you care about. The point is not the winner — it is the discipline of running a fair evaluation and writing the report someone else can act on.
Objectives
- 01Choose a task specific enough to be measurable.
- 02Design an evaluation that is fair across providers.
- 03Report cost, latency, quality, and failure modes.
- 04Pick a winner with a written decision rule.
Key Ideas
- A bake-off without a fixed test set is theater.
- The winner today is the loser after the next model release; decision rules survive both.
Lab
Select one task from your work (structured extraction, classification, summarization, or code generation). Assemble a fixed test set of at least 30 examples with known correct outputs. Run the task on GPT-4o, Claude Sonnet, Gemini, and one local model (Llama or Qwen, quantized). For each: record latency (p50 and p95), cost per call, and quality on a rubric. Publish a report that recommends one model for the task with a decision rule stating when that recommendation would change.
Deliverables
- — Test set (30+ examples) with ground truth.
- — Reproducible run script for all four providers.
- — Report with latency, cost, and quality tables plus qualitative failure notes.
- — Explicit decision rule and threshold for revisiting the choice.
Rubric
- — Fairness — same prompt, same test set, same rubric across providers.
- — Honesty — failure modes reported, not hidden.
- — Cost accuracy — token accounting matches provider invoices.
- — Actionability — someone reading the report could pick and ship.