Summary
The RAG lab: 20 documents, 10 questions, a measured system.
Key Ideas
- A RAG system is not built until it is measured.
- Corpus you know beats a corpus you don't — you can spot wrong answers.
Lab
Choose a corpus you know well (a set of books, your own writing, a company's docs). Ingest 20+ documents. Build a RAG pipeline: chunk → embed → index → retrieve (hybrid) → rerank → generate with citations. Author 10 questions with reference answers. Measure retrieval recall@5 and answer correctness. Report the numbers.
Deliverables
- — Ingest pipeline and populated vector store.
- — Query pipeline with hybrid retrieval, reranker, and cited generation.
- — 10 questions with reference answers.
- — Report with retrieval and generation metrics separately.
Rubric
- — Pipeline completeness — every stage (chunk, embed, retrieve, rerank, generate, cite) implemented.
- — Evaluation discipline — retrieval and generation reported separately.
- — Citation validity — every citation clicks through to a real chunk.
- — Honesty — failure modes reported, not smoothed over.