RAG Reranking

RAG reranking is a second ranking pass that reorders retrieved chunks by how well they actually answer the question. Vector search is fast but approximate; a reranker is slower but much more precise. Combining them gives the best of both.

Bi-Encoders vs Cross-Encoders

Embedding models are bi-encoders: they embed the query and each document separately, then compare vectors. That is fast because document vectors are pre-computed.

A cross-encoder reads the query and a document together and outputs one relevance score. It understands the relationship much better, but must run once per pair, so you only use it on a short list.

The Retrieve-then-Rerank Pattern

• Retrieve a generous candidate set, for example the top 20 to 50 chunks.
• Score each (question, chunk) pair with a cross-encoder.
• Keep the top 3 to 5 by reranker score for the prompt.

Reranking with sentence-transformers

Install with pip install sentence-transformers. BAAI/bge-reranker-v2-m3 is a strong open-source multilingual reranker; bge-reranker-base is smaller and faster.

from sentence_transformers import CrossEncoder

reranker = CrossEncoder("BAAI/bge-reranker-base")
question = "How do I reset my password?"
candidates = [
    "Passwords must contain at least 12 characters.",
    "To reset your password, click 'Forgot password' and follow the email link.",
    "Our password manager integration supports 1Password.",
]
scores = reranker.predict([(question, c) for c in candidates])
for s, c in sorted(zip(scores, candidates), reverse=True):
    print(round(float(s), 3), c)

Sample output:
0.981 To reset your password, click 'Forgot password' and follow the email link.
0.087 Passwords must contain at least 12 characters.
0.012 Our password manager integration supports 1Password.

All three candidates mention passwords, so a vector search might rank them closely. The cross-encoder clearly picks the one that answers the question.

Plugging It into the Pipeline

def retrieve_reranked(question, fetch_k=20, top_n=4):
    hits = retrieve(question, k=fetch_k)            # from the from-scratch lesson
    docs = [doc for doc, meta, _ in hits]
    scores = reranker.predict([(question, d) for d in docs])
    ranked = sorted(zip(scores, hits), key=lambda x: x[0], reverse=True)
    return [h for _, h in ranked[:top_n]]

(returns the 4 best chunks out of 20 candidates)

Reranking Tips

• Rerankers add latency; on CPU keep candidates under about 50.
• Hosted rerankers (Cohere, Jina, Voyage) exist if you cannot run models locally.
• An LLM can also rerank by scoring relevance, but it is slower and costlier.
• Measure: reranking usually boosts precision noticeably, but verify on your data.

Frequently asked questions

What is a reranker in RAG?

A model, usually a cross-encoder, that scores how relevant each retrieved chunk is to the query so the best ones can be placed in the prompt.

Does reranking slow down RAG?

It adds some latency, typically tens to hundreds of milliseconds for 20 to 50 candidates on a GPU, more on CPU.

Can I use Ollama for reranking?

Ollama focuses on chat and embedding models. Common practice is to run rerankers with sentence-transformers, or prompt an LLM in Ollama to score relevance.