Hugging Face Project: Build a Mini RAG Assistant

This Hugging Face project ties the course together by building a mini RAG (retrieval-augmented generation) assistant. It answers questions about your own notes by first finding the most relevant passages with embeddings, then asking an open-weight LLM to answer using only those passages.

Step 1: The Knowledge Base

notes = [
    "Our office is open Monday to Friday, 9am to 6pm Gulf Standard Time.",
    "Refunds are processed within 7 business days of approval.",
    "Support can be reached at the in-app chat or by email.",
    "Premium plans include priority support and a dedicated account manager.",
]
(no output — 4 passages defined)

Step 2: Embed and Retrieve

from sentence_transformers import SentenceTransformer

embedder = SentenceTransformer("sentence-transformers/all-MiniLM-L6-v2")
note_emb = embedder.encode(notes)

def retrieve(question, k=2):
    q = embedder.encode([question])
    scores = embedder.similarity(q, note_emb)[0]
    top = scores.argsort(descending=True)[:k]
    return [notes[i] for i in top]

print(retrieve("How long does a refund take?"))
['Refunds are processed within 7 business days of approval.', 'Support can be reached at the in-app chat or by email.']
(sample — second result may differ)

Step 3: Generate a Grounded Answer

from transformers import pipeline

llm = pipeline("text-generation", model="Qwen/Qwen2.5-0.5B-Instruct")

def answer(question):
    context = "\n".join("- " + p for p in retrieve(question))
    messages = [
        {"role": "system", "content": "Answer only from the context. If the answer is not there, say you don't know."},
        {"role": "user", "content": "Context:\n" + context + "\n\nQuestion: " + question},
    ]
    out = llm(messages, max_new_tokens=80, do_sample=False)
    return out[0]["generated_text"][-1]["content"]

print(answer("When is the office open?"))
print(answer("Do you sell gift cards?"))
The office is open Monday to Friday, from 9am to 6pm Gulf Standard Time.
I don't know.
(sample output — generated text varies)

do_sample=False makes answers more stable. The system message and the "say you don't know" instruction reduce made-up answers, although small models still sometimes ignore it.

Step 4: Put It in a Gradio App

import gradio as gr

gr.Interface(fn=answer, inputs="text", outputs="text", title="Ask Our Docs").launch()
* Running on local URL:  http://127.0.0.1:7860

Exercises

  • Load your notes from a CSV with the datasets library instead of a Python list.
  • Swap the embedder for a bge model and compare which passages are retrieved.
  • Load a 7B instruct model in 4-bit and compare answer quality.
  • Replace the local LLM with InferenceClient and measure the speed difference.
  • Deploy the Gradio app to a Space and keep any token in Secrets.
  • Fine-tune the LLM with LoRA on example question-answer pairs from your domain.

Frequently asked questions

What is RAG in Hugging Face?

Retrieval-augmented generation: you retrieve relevant documents with embeddings and pass them to an LLM as context so its answer is grounded in your data.

Do I need a vector database for RAG?

Not for small projects; NumPy arrays or tensors are enough. For large collections, a vector database or FAISS index makes search fast.

What are good beginner Hugging Face projects?

A sentiment analyzer Space, a semantic search over your notes, a fine-tuned classifier for support tickets, and a small RAG chatbot are all excellent first projects.