Summary
Retrieval returns 50-200 candidates; the generator can only use 5-20. A reranker sits between: a smaller model that scores each candidate against the query with more expressiveness than a raw dot product. Adding a reranker is the highest-return improvement most RAG systems have not made.
Objectives
- 01Distinguish retrieval, ranking, and reranking.
- 02Name the two dominant reranker types.
- 03State the latency cost of adding a reranker.
The Lesson
The stage
Retrieval returns candidates by rough similarity. The reranker scores each candidate against the query with a cross-encoder that reads query and document jointly. Higher quality than a dot product; slower.
Choices
Cohere Rerank, Voyage rerank, or open-source (BGE, Jina rerankers). Latency: 50-200ms for 50 candidates. Nearly always worth it. LLM-as-reranker works but costs more; use for the top 10-20 after a cheaper reranker.
Key Ideas
- Reranking is the highest-return upgrade for most RAG systems.
- Cross-encoders beat dot products at a cost most systems can pay.