Summary
Instrumentation as a first-class practice: dashboards, alerts, and one measurable optimization based on the data. The lab that separates engineers from prompt tinkerers.
Objectives
- 01Instrument latency at p50, p95, p99.
- 02Instrument cost per request and daily spend.
- 03Set alerts with defensible thresholds.
- 04Ship one optimization driven by the data.
Key Ideas
- You cannot optimize what you have not measured for a week.
- The largest bill line is almost never where you assumed.
Lab
Take an AI service (yours or a peer's) that handles at least 100 requests per day. Add instrumentation for latency (p50/p95/p99) and cost (per-request and daily total), broken down by endpoint and model. Ship dashboards and alerts (e.g., p95 above 3s, daily cost above budget). After one week of data, identify the largest cost or latency line, ship one optimization, and measure the delta. Publish the before/after data and the write-up.
Deliverables
- — Instrumentation code and dashboards.
- — Alert rules with justified thresholds.
- — One week of collected data.
- — One optimization with measured before/after impact.
Rubric
- — Instrumentation coverage — every request produces cost and latency data.
- — Alert quality — thresholds are defensible, not arbitrary.
- — Data-driven change — the optimization targets what the data showed.
- — Measurement — the before/after delta is reported honestly.