Summary
A working retrieval-augmented-generation system with retrieval and generation evaluated separately. The lab that separates people who have read about RAG from people who have shipped it.
Objectives
- 01Ingest a corpus, chunk it, embed it, index it.
- 02Retrieve with hybrid search and rerank.
- 03Generate answers with citations.
- 04Evaluate retrieval quality and answer quality separately.
Key Ideas
- Retrieval and generation are separate systems; evaluate them separately.
- A citation you cannot click is not a citation.
Lab
Build a RAG system over a corpus you know well (200+ documents — company docs, research papers, or a book series). Use hybrid search (BM25 + vector), a reranker, and a generation step that produces cited answers. Author a test set of 50 questions with reference answers. Report retrieval metrics (recall@k, MRR) separately from answer metrics (faithfulness, correctness). Deploy the system behind a simple UI.
Deliverables
- — Deployed RAG endpoint or app.
- — 50-question test set with reference answers.
- — Evaluation report with retrieval and generation metrics separately.
- — Architecture diagram and design decisions memo.
Rubric
- — Retrieval quality — recall@10 above 0.8 on the test set.
- — Generation quality — faithfulness above 0.9; hallucinations tracked.
- — Separation — retrieval and generation metrics reported independently.
- — Deployment — the system runs; someone else can query it.