RAG Project: Chat with Your PDFs

This RAG project builds a chat-with-your-PDFs app that runs entirely on your computer. Drop PDFs into a folder, index them, and ask questions in a chat loop that remembers follow-ups and cites file names and page numbers. It reuses ideas from every lesson.

Setup

pip install ollama chromadb pypdf
ollama pull nomic-embed-text
ollama pull llama3.2
mkdir pdfs   # put your PDF files here

(packages installed, models pulled)

Step 1: Extract, Chunk and Index PDFs

import pathlib, ollama, chromadb
from pypdf import PdfReader

col = chromadb.PersistentClient(path="./pdf_db").get_or_create_collection(
    "pdfs", metadata={"hnsw:space": "cosine"})

def embed(texts):
    return ollama.embed(model="nomic-embed-text", input=texts)["embeddings"]

def chunk(text, size=800, overlap=150):
    return [text[i:i + size] for i in range(0, len(text), size - overlap) if text[i:i + size].strip()]

def index_pdfs(folder="pdfs"):
    total = 0
    for pdf in pathlib.Path(folder).glob("*.pdf"):
        for page_no, page in enumerate(PdfReader(pdf).pages, start=1):
            pieces = chunk(page.extract_text() or "")
            if not pieces:
                continue
            col.upsert(
                ids=[f"{pdf.name}-p{page_no}-c{i}" for i in range(len(pieces))],
                documents=pieces,
                embeddings=embed(pieces),
                metadatas=[{"file": pdf.name, "page": page_no}] * len(pieces),
            )
            total += len(pieces)
    return total

print("Indexed", index_pdfs(), "chunks")

Sample output:
Indexed 212 chunks

Chunking per page keeps page numbers exact for citations. Pages with no extractable text (scans) are skipped; add OCR if you need them.

Step 2: Retrieval with Query Rewriting

def search(query, k=5):
    r = col.query(query_embeddings=embed([query]), n_results=k)
    return list(zip(r["documents"][0], r["metadatas"][0]))

def standalone(history, question):
    if not history:
        return question
    convo = "\n".join(f"{m['role']}: {m['content']}" for m in history[-4:])
    r = ollama.chat(model="llama3.2", options={"temperature": 0}, messages=[{"role": "user",
        "content": f"Chat:\n{convo}\n\nRewrite as a standalone search query, only the query: {question}"}])
    return r["message"]["content"].strip()

(no output: defines retrieval helpers)

Step 3: The Chat Loop with Citations

SYSTEM = ("You answer questions about the user's PDFs using only the numbered context. "
          "Cite like [1]. If the answer is not in the context, say you don't know.")
history = []

while (q := input("\nYou: ").strip()) not in {"", "exit"}:
    hits = search(standalone(history, q))
    context = "\n\n".join(f"[{i}] {doc}" for i, (doc, _) in enumerate(hits, 1))
    messages = [{"role": "system", "content": SYSTEM}, *history[-6:],
                {"role": "user", "content": f"Context:\n{context}\n\nQuestion: {q}"}]
    print("Bot: ", end="")
    reply = ""
    for part in ollama.chat(model="llama3.2", messages=messages, stream=True):
        print(part["message"]["content"], end="", flush=True)
        reply += part["message"]["content"]
    print("\nSources:", ", ".join(f"[{i}] {m['file']} p.{m['page']}" for i, (_, m) in enumerate(hits, 1)))
    history += [{"role": "user", "content": q}, {"role": "assistant", "content": reply}]

Sample session:
You: What is the refund window?
Bot: Customers can request a refund within 30 days of delivery [1].
Sources: [1] terms.pdf p.3, [2] terms.pdf p.4, ...
You: Does that include sale items?
Bot: No, sale items are final and cannot be refunded [2].
Sources: [1] terms.pdf p.4, [2] terms.pdf p.4, ...

History stores the plain question, not the long context, so it stays small. The rewrite step makes "Does that include sale items?" searchable.

Ideas to Extend the Project

• Add the cross-encoder reranker: fetch 20, keep the best 5.
• Add BM25 and fuse with RRF for exact terms.
• Build a test set and track hit rate and faithfulness.
• Wrap it in Streamlit or FastAPI with a file upload button.
• Rebuild it with LangChain or swap Chroma for pgvector or Qdrant.

Frequently asked questions

Can I chat with PDFs offline?

Yes. With Ollama for models and Chroma for storage, nothing leaves your computer once the models are downloaded.

Why does my PDF chatbot give wrong page numbers?

Usually because chunks span pages. Chunk each page separately, as in this project, and store the page in metadata.

How do I handle scanned PDFs?

Run OCR first (for example with Tesseract or a layout-aware parser like Docling) to get text, then index as normal.