RAG Citing Sources

RAG citing sources lets users check where an answer came from, builds trust, and makes hallucinations easy to spot. The trick is to keep metadata with every chunk and design the prompt so the model refers to sources by number.

Number the Context

hits = [
    ("Annual leave is 20 days.", {"source": "handbook.pdf", "page": 4}),
    ("Unused leave up to 5 days can be carried over.", {"source": "handbook.pdf", "page": 5}),
]
context = "\n\n".join(
    f"[{i}] ({m['source']} p.{m['page']}) {text}" for i, (text, m) in enumerate(hits, start=1)
)
print(context)

[1] (handbook.pdf p.4) Annual leave is 20 days.

[2] (handbook.pdf p.5) Unused leave up to 5 days can be carried over.

A Prompt That Requires Citations

SYSTEM = """You answer using ONLY the numbered context.
After each sentence, cite the supporting source like [1] or [1][2].
If the context does not contain the answer, reply exactly: I don't know."""

Sample model answer:
You get 20 days of annual leave [1], and up to 5 unused days can be carried over [2].

Structured Citations

For a UI that shows clickable sources, ask for JSON with Ollama structured outputs and a Pydantic schema:

from pydantic import BaseModel
import ollama

class Answer(BaseModel):
    answer: str
    citations: list[int]

resp = ollama.chat(
    model="llama3.2",
    messages=[{"role": "system", "content": SYSTEM},
              {"role": "user", "content": f"{context}\n\nQuestion: How much leave do I get?"}],
    format=Answer.model_json_schema(),
    options={"temperature": 0},
)
ans = Answer.model_validate_json(resp["message"]["content"])
print(ans.answer, [hits[i - 1][1] for i in ans.citations])

Sample output:
You get 20 days of annual leave. [{'source': 'handbook.pdf', 'page': 4}]

Verify the Citations

import re

def check_citations(text, n_sources):
    cited = {int(x) for x in re.findall(r"\[(\d+)\]", text)}
    return {"cited": sorted(cited), "invalid": sorted(c for c in cited if not 1 <= c <= n_sources)}

print(check_citations("Leave is 20 days [1]. Carry over 5 [2]. Bonus [7].", 2))

{'cited': [1, 2, 7], 'invalid': [7]}

Invalid numbers are a red flag. You can also check that cited chunks really support the sentence using an LLM judge, covered in the evaluation lesson.

Citation Best Practices

• Store source, page, section and URL in metadata at indexing time.
• Show the quoted chunk text when users hover a citation.
• Deduplicate sources before display.
• Treat "no citation" answers as lower confidence.

Frequently asked questions

How do I make an LLM cite sources in RAG?

Number each context chunk, instruct the model to cite numbers after each claim, and map the numbers back to the chunk metadata in your code.

Can RAG citations be wrong?

Yes. Models sometimes cite the wrong chunk or invent numbers. Validate citation numbers and spot-check support with evaluation.

Should I show page numbers?

Yes, when available. Page or section references let users verify answers quickly.