Reranking
Scenario
You search your docs for 'how to cancel subscription'. The vector database returns 20 chunks. The actual cancellation instructions are at position 14. Your RAG prompt only feeds the top 5 to the LLM to save tokens. The model never sees the answer. Before you read on, predict: does the vector database have a bug?
Think about how vector similarity works — then read on to understand why the right answer ends up at position 14.
Mental model
Vector search is like scanning a library with a metal detector — fast but imprecise. Reranking is having an expert sit down with the 20 items the detector flagged and carefully sort them by true relevance.
Standard embeddings compress a whole paragraph into a single point in vector space. This compression loses nuance. So fast vector search (bi-encoders) often misses the subtle difference between a good match and a great match. The fix is a two-stage pipeline: first use fast vector search to fetch the top 50 'okay' results, then pass those 50 to a reranker (cross-encoder) that reads each chunk alongside the query to score true relevance.
Deep dive
Vector similarity is fast because it is precomputed. You embed chunks once at ingest time, then calculate distances at query time — milliseconds for millions of chunks. But this speed has a cost: bi-encoders embed the query and chunk in isolation. They never compare them together. This means they miss nuanced relevance signals — a chunk that uses slightly different words from the query may rank poorly even if it directly answers the question.
Fast vector search embeds query and chunk separately, which often misses nuanced relevance.
The consequence: the right answer ends up ranked at position 14 instead of position 1. If your RAG prompt only takes the top 5 chunks to stay under the token limit, the LLM never sees the answer. The vector database is working correctly — it just cannot measure relevance with the precision you need.
If the correct answer is ranked at position 14, a top-5 RAG pipeline fails — through no fault of the model.
A reranker (usually a cross-encoder) reads the query and the candidate chunk at the same time in one forward pass. This joint reading lets it understand the precise relationship between the specific question and the specific text. It outputs a relevance score from 0 to 1. This is far more accurate than a bi-encoder similarity score.
Cross-encoders read the query and chunk together — which is why they score relevance much more accurately.
If cross-encoders are so accurate, why not use them for the whole database? They are too slow. Running a cross-encoder against 1 million chunks would take minutes per query. Vector search takes milliseconds. So you combine both: speed first, accuracy second.
Cross-encoders are too slow for millions of documents. Use them only on a small shortlist from vector search.
The two-stage pipeline: **Step 1** — fast vector search retrieves 50 candidate chunks. **Step 2** — a reranker scores all 50 against the query. **Step 3** — sort by reranker score. **Step 4** — feed the new top 5 to the LLM. This gives you the speed of vector search and the precision of a cross-encoder.
Pipeline: Query → Vector search (top 50) → Rerank (top 5) → LLM prompt.
Code examples
Two-stage retrieval using the Cohere Rerank API
import cohere
co = cohere.Client('YOUR_API_KEY')
query = 'How to cancel subscription'
# Stage 1: fast vector search — returns 20 candidate chunks
initial_results = vector_db.search(query, k=20) # list of text strings
# Stage 2: rerank those 20 chunks for deep relevance
rerank_response = co.rerank(
model='rerank-english-v3.0',
query=query,
documents=initial_results,
top_n=5, # keep the best 5 for the LLM prompt
)
final_chunks = [initial_results[r.index] for r in rerank_response.results]You do not rerank the entire database — only the shortlist from vector search. The reranker runs 20 deep comparisons (fast) and surfaces the most relevant 5. The correct answer — which was at position 14 in similarity order — now appears in the top 5.
Key points
- Nearest ≠ most relevant: A vector database returns the geometrically closest points, which are not always the logically most relevant answers to the question.
- Bi-encoder vs cross-encoder: Vector search uses bi-encoders (fast, isolated). Reranking uses cross-encoders (slower, simultaneous, accurate).
- Two-stage retrieval: Always use both. Vector search filters millions down to 50. Reranking filters 50 down to the 5 you actually send to the LLM.
- Cost and latency: Reranking adds a network hop (typically 100–200ms). The accuracy gain is usually worth it for high-value queries.
Common mistakes
- Trying to rerank the whole database: You cannot run a cross-encoder against 100,000 documents at query time. It must be used only on the small shortlist (top 50–100) returned by fast vector search.
- Using reranking when initial retrieval fails completely: Reranking only reorders what it is given. If the vector search missed the relevant document entirely — it was ranked at position 500 and you only fetched top 20 — the reranker cannot magically find it. Expand your initial fetch size.
Glossary
- bi-encoder
- The embedding model used for vector search — it embeds the query and each document chunk separately, then compares them. Fast but loses nuance.
- reranker
- A specialized model that reads the query and a candidate document together at the same time to score how well they match — slower but much more accurate.
- cross-encoder
- The type of model used for reranking — it processes the query and document jointly in one forward pass, allowing it to catch subtle relevance signals that bi-encoders miss.
Recall questions
- Explain what reranking does and why it is useful, as if to a classmate.
- Why is vector search (bi-encoder) less accurate than a reranker (cross-encoder)?
- If rerankers are more accurate, why not replace vector search with a reranker entirely?
- Your RAG pipeline fails: the correct answer exists in the database at position 40 but you only fetch the top 20. Will adding a reranker fix this?
Questions & answers
Your RAG pipeline fetches 5 chunks from the vector database. The LLM's answer is wrong, but you manually verify the correct chunk exists at position 15 in the database. You cannot increase the LLM context window. How do you fix this?
Implement two-stage retrieval. Expand the initial vector search to fetch 20–50 chunks, then pass those through a reranker. The reranker will score them for deep relevance and move the correct chunk from position 15 into the top 5, so it fits in the context window.
Approach: Identify that the vector search finds the document but ranks it too low. Reranking is the canonical solution for the 'right doc, wrong position' problem.
You are designing a low-latency Q&A system. A colleague suggests using a cross-encoder to directly search the 500,000-document corpus to maximize accuracy. Why is this a bad idea?
A cross-encoder must run the query against every document individually at query time. Running 500,000 deep neural network passes sequentially takes minutes or hours per query — it violates the low-latency requirement. The correct architecture is a fast bi-encoder vector search first (milliseconds), then a cross-encoder reranker on the resulting shortlist (a few hundred milliseconds).
Approach: Highlight the catastrophic performance impact of running a cross-encoder at full corpus scale. Cross-encoders must always be gated behind a fast first-stage retrieval.
Continue learning
Previous: Embed & Retrieve Top-k
Next: Evaluation & Test Sets