RAG Vector Databases

RAG vector databases store embeddings and find the nearest ones to a query vector quickly. A plain Python loop works for a few thousand chunks, but a vector database adds fast approximate search, persistence, metadata filtering and scaling to millions of vectors.

What a Vector Database Does

• Stores each vector with its original text and metadata.
• Builds an ANN (approximate nearest neighbour) index, commonly HNSW, for fast search.
• Supports distance metrics: cosine, dot product or Euclidean (L2).
• Filters by metadata, for example only documents for one customer.
• Handles updates, deletes and persistence.

Chroma

Chroma is an open-source, Python-friendly database that can run embedded in your app. Install with pip install chromadb.

import chromadb

client = chromadb.PersistentClient(path="./chroma_data")
col = client.get_or_create_collection("notes", metadata={"hnsw:space": "cosine"})
col.add(ids=["a", "b"],
        documents=["Cats sleep a lot", "Python is a programming language"],
        embeddings=[[0.9, 0.1, 0.0], [0.0, 0.2, 0.9]])
print(col.query(query_embeddings=[[0.8, 0.2, 0.1]], n_results=1)["documents"])

[['Cats sleep a lot']]

Tiny 3-number vectors keep the example readable; real ones come from an embedding model. If you omit embeddings, Chroma uses a built-in default embedding function.

pgvector (PostgreSQL)

pgvector is an open-source extension that adds a vector column type to PostgreSQL. It is ideal when your app already uses Postgres, because vectors live next to your normal data and you can join and filter with SQL.

CREATE EXTENSION IF NOT EXISTS vector;

CREATE TABLE chunks (
  id        bigserial PRIMARY KEY,
  content   text,
  source    text,
  embedding vector(768)
);
CREATE INDEX ON chunks USING hnsw (embedding vector_cosine_ops);

-- <=> is cosine distance; smaller is closer
SELECT content, source FROM chunks
ORDER BY embedding <=> '[0.01, -0.02, ...]'
LIMIT 4;

Sample output:
 content                                  | source
------------------------------------------+-------------
 Employees receive 20 days of paid leave  | handbook.pdf
 ...

Qdrant

Qdrant is an open-source vector database written in Rust, built for performance and rich filtering. Run it with Docker or use in-memory mode for testing (pip install qdrant-client).

from qdrant_client import QdrantClient
from qdrant_client.models import Distance, VectorParams, PointStruct

client = QdrantClient(":memory:")
client.create_collection("notes", vectors_config=VectorParams(size=3, distance=Distance.COSINE))
client.upsert("notes", points=[
    PointStruct(id=1, vector=[0.9, 0.1, 0.0], payload={"text": "Cats sleep a lot"}),
    PointStruct(id=2, vector=[0.0, 0.2, 0.9], payload={"text": "Python is a language"}),
])
hits = client.query_points("notes", query=[0.8, 0.2, 0.1], limit=1).points
print(hits[0].payload["text"], round(hits[0].score, 3))

Sample output:
Cats sleep a lot 0.985

Which One Should You Choose?

• Chroma: easiest for prototypes, notebooks and small to medium apps.
• pgvector: already on PostgreSQL, want SQL joins, transactions and one database.
• Qdrant: large collections, heavy filtering, high throughput, built-in hybrid search.
• Others worth knowing: Milvus, Weaviate, LanceDB, OpenSearch and Elasticsearch.

Frequently asked questions

Do I need a vector database for RAG?

Not for small datasets: NumPy and a few thousand vectors work fine. A vector database becomes useful for persistence, filtering and scale.

Is pgvector good enough for production?

Yes. With HNSW indexes pgvector handles millions of vectors well, and many teams prefer keeping everything in PostgreSQL.

Which distance metric should I use?

Use the metric your embedding model was trained for; cosine is the safe default for most text embedding models.