RAG Pipeline from Scratch with Ollama and Chroma

In this lesson you build a complete RAG pipeline from scratch in about 50 lines of Python, using only the ollama and chromadb packages. No framework hides anything, so you will see exactly how each step works.

Setup

pip install ollama chromadb
ollama pull nomic-embed-text
ollama pull llama3.2

Successfully installed chromadb-... ollama-...

Step 1: Chunk and Index

import ollama, chromadb

EMBED_MODEL, CHAT_MODEL = "nomic-embed-text", "llama3.2"
client = chromadb.PersistentClient(path="./rag_db")
col = client.get_or_create_collection("kb", metadata={"hnsw:space": "cosine"})

def chunk(text, size=600, overlap=100):
    return [text[i:i + size] for i in range(0, len(text), size - overlap)]

def embed(texts):
    return ollama.embed(model=EMBED_MODEL, input=texts)["embeddings"]

def index_file(path):
    text = open(path, encoding="utf-8").read()
    chunks = chunk(text)
    col.upsert(
        ids=[f"{path}-{i}" for i in range(len(chunks))],
        documents=chunks,
        embeddings=embed(chunks),
        metadatas=[{"source": path, "chunk": i} for i in range(len(chunks))],
    )
    return len(chunks)

print(index_file("company_faq.txt"), "chunks indexed")

Sample output:
14 chunks indexed

upsert with deterministic IDs means running the indexer again updates chunks instead of duplicating them. We pass our own embeddings so Chroma uses the Ollama model, not its default.

Step 2: Retrieve

def retrieve(question, k=4):
    res = col.query(query_embeddings=embed([question]), n_results=k,
                    include=["documents", "metadatas", "distances"])
    return list(zip(res["documents"][0], res["metadatas"][0], res["distances"][0]))

for doc, meta, dist in retrieve("What are your support hours?", k=2):
    print(round(dist, 3), meta["source"], doc[:50])

Sample output:
0.214 company_faq.txt Support is available Monday to Saturday, 9 am
0.398 company_faq.txt For urgent issues outside support hours, email

Step 3: Augment and Generate

SYSTEM = ("Answer the question using only the context. "
          "If the context does not contain the answer, say \"I don't know.\" "
          "Cite sources as [n].")

def answer(question):
    hits = retrieve(question)
    context = "\n\n".join(f"[{i + 1}] {doc}" for i, (doc, meta, _) in enumerate(hits))
    resp = ollama.chat(model=CHAT_MODEL, options={"temperature": 0}, messages=[
        {"role": "system", "content": SYSTEM},
        {"role": "user", "content": f"Context:\n{context}\n\nQuestion: {question}"},
    ])
    return resp["message"]["content"], [m for _, m, _ in hits]

text, sources = answer("What are your support hours?")
print(text)

Sample output:
Support is available Monday to Saturday, 9 am to 6 pm [1]. For urgent issues outside these hours, you can email the on-call team [2].

What You Just Built

• chunk: fixed-size chunks with overlap
• embed: local embeddings via Ollama
• col.upsert / col.query: Chroma vector storage and search
• answer: a grounded prompt with numbered context and an "I don't know" rule
• A list of sources returned alongside the answer for citations

Every RAG framework does these same steps. Improving them (better chunking, reranking, hybrid search) is what the rest of the tutorial covers.

Frequently asked questions

Can I build RAG without LangChain or LlamaIndex?

Yes. As this lesson shows, a basic pipeline is an embedding call, a vector database query and a chat call. Frameworks add convenience and integrations, not magic.

Why use temperature 0 for RAG answers?

Lower temperature makes the model stick closer to the context and give consistent answers, which is what you want for factual Q&A.

How do I index PDFs in this pipeline?

Extract text page by page with a library like pypdf, then chunk each page and store the page number in metadata.