Summary
A domain-specific coding assistant with real evaluation. Not a wrapper around Copilot — an assistant tuned for one codebase, with tests that measure whether it helps.
Objectives
- 01Build an assistant for one defined codebase.
- 02Provide it with domain context via retrieval or fine-tune.
- 03Instrument suggestions and acceptance rates.
- 04Evaluate correctness with tests, not vibes.
Key Ideas
- A coding assistant lives or dies on retrieval quality over the codebase.
- The only honest metric is 'did the tests pass after the suggestion?'
Lab
Build a coding assistant specialized for one codebase you know (an open-source library or a defined internal repo). Provide it with codebase context via retrieval over source, docs, and issue history. Ship it as a CLI or IDE extension. Instrument suggestion generation, acceptance, and rejection. Evaluate on 30 real tasks pulled from the repository's issue history: does the suggestion make tests pass? Report acceptance rate and pass rate.
Deliverables
- — Working assistant (CLI or IDE extension).
- — Retrieval index over source, docs, and history.
- — Instrumentation logs with suggestion/acceptance/pass data.
- — Evaluation report on 30 real tasks.
Rubric
- — Specialization — measurably better than a generic assistant on this codebase.
- — Instrumentation — you can answer 'is it working?' with data.
- — Correctness — evaluated with actual tests, not judgment calls.
- — Deployability — installable by another engineer without your help.