RAG Similarity Search

RAG similarity search is the retrieval step: given a query vector, find the stored chunks whose vectors are closest. This lesson implements it by hand so you understand what vector databases do internally.

Distance Metrics

• Cosine similarity: angle between vectors; ignores length. Higher is more similar.
• Dot product: equals cosine when vectors are normalized to length 1. Higher is more similar.
• Euclidean (L2) distance: straight-line distance. Lower is more similar.
• Cosine distance = 1 - cosine similarity, which is what many databases report.

Brute-Force Top-k Search

import math

def cosine(a, b):
    return sum(x * y for x, y in zip(a, b)) / (math.hypot(*a) * math.hypot(*b))

index = [
    ("Refunds take 7 days", [0.9, 0.1, 0.1]),
    ("Shipping is free over $50", [0.2, 0.9, 0.1]),
    ("Refund requests need a receipt", [0.8, 0.2, 0.2]),
    ("We are hiring engineers", [0.1, 0.1, 0.9]),
]

def search(qvec, k=2):
    scored = [(cosine(qvec, v), text) for text, v in index]
    return sorted(scored, reverse=True)[:k]

for score, text in search([0.85, 0.15, 0.1]):
    print(round(score, 3), text)

0.992 Refunds take 7 days
0.99 Refund requests need a receipt

This compares the query to every vector, which is exact but slow for millions of items. Vector databases use ANN indexes like HNSW to skip most comparisons while staying very accurate.

Score Thresholds

Top-k always returns k results, even if none are relevant. A threshold drops weak matches so the model can say "I don't know":

def search_threshold(qvec, k=4, min_score=0.9):
    return [(s, t) for s, t in search(qvec, k) if s >= min_score]

print([(round(s, 3), t) for s, t in search_threshold([0.1, 0.1, 0.95])])
print(search_threshold([0.5, 0.5, 0.5]))

[(1.0, 'We are hiring engineers')]
[]

Thresholds depend on the embedding model, so calibrate them by looking at scores for known good and bad matches.

Metadata Filters

Filtering narrows the search before or during ranking, for example to one product, language, date range or user's permissions. In Chroma you pass where={"product": "pro"}; in pgvector you add a WHERE clause; in Qdrant a Filter object.

MMR: Diverse Results

Maximal Marginal Relevance picks results that are relevant to the query but not too similar to each other. It avoids filling the prompt with three overlapping chunks that say the same thing. Most frameworks offer it as a search option.

Frequently asked questions

What is k in RAG retrieval?

The number of chunks returned for each query. Typical values are 3 to 8, depending on chunk size and the model's context window.

Cosine similarity or dot product?

If vectors are normalized they give the same ranking. Use whatever your embedding model recommends; cosine is the safe default.

Why does retrieval return irrelevant chunks?

Top-k always returns something. Add a score threshold, improve chunking, use hybrid search or rerank the results.