Summary
A statistically defensible A/B test on an AI product change. Design, run, analyze, decide — with novelty effects, power calculations, and a written decision rule.
Objectives
- 01Design an A/B test with a defensible primary metric.
- 02Compute the sample size and expected duration.
- 03Run the test and monitor for novelty effects.
- 04Analyze results and decide with a written rule.
Key Ideas
- The decision rule is written before the data, not after.
- Novelty effects invert cleanly in week two; run the full duration.
Lab
Choose an AI product change (a prompt revision, a model swap, a retrieval improvement). Author a design doc: primary metric, minimum detectable effect, sample size, guardrail metrics, novelty-effect plan. Run the test long enough to hit statistical power (or document why you stopped early). Analyze with a proper significance test. Write the decision: ship, kill, or extend. Publish the design doc, the raw data, the analysis, and the memo.
Deliverables
- — Design doc with power calculation and guardrail metrics.
- — Test run for the pre-committed duration.
- — Statistical analysis with confidence intervals.
- — Written decision memo with rationale.
Rubric
- — Pre-registration — the analysis plan is committed before the data lands.
- — Statistical rigor — power computed, guardrails watched.
- — Novelty handling — the analysis distinguishes durable effect from novelty.
- — Decision discipline — the rule is followed even when the result is uncomfortable.