Prompt Engineering for RAG

Prompt engineering for RAG (retrieval-augmented generation) is about making the model answer from documents you retrieved, not from memory. A good RAG prompt clearly separates sources, demands citations and allows "I don't know".

A Grounded Answer Prompt

Answer the question using only the sources below.
Cite sources like [1]. If the sources do not contain the answer, say
"I couldn't find that in the documentation."

<sources>
[1] Annual leave: full-time staff receive 22 days per year.
[2] Leave requests must be submitted 2 weeks in advance via the HR portal.
</sources>

Question: How much notice do I need to give for leave?
Sample response:
You need to submit leave requests 2 weeks in advance through the HR portal [1].

(Note: the correct citation is [2]. Checking citations automatically is part of evaluation.)

The sample shows a real failure mode: a wrong citation. RAG prompts reduce hallucinations but do not remove them, so evaluate citations as well as answers.

RAG Prompt Tips

  • Number or label each chunk so citations are checkable.
  • Put retrieved text before the question.
  • Tell the model what to do when sources conflict (prefer newest, or mention both).
  • Include metadata like titles and dates in each chunk.
  • Retrieve fewer, better chunks rather than many noisy ones.

Handling Missing Information

Question: What is the parental leave policy?
(sources only mention annual leave)
Sample response:
I couldn't find that in the documentation.

Query Rewriting

Users ask short, vague questions. A small first prompt can rewrite them into a better search query before retrieval:

Rewrite the user's question as a standalone search query.
Conversation: "How much leave do I get?" ... "and for part-timers?"
Sample response:
annual leave entitlement for part-time staff

Frequently asked questions

Does RAG stop hallucinations?

It greatly reduces them when retrieval finds the right text, but the model can still misread or misquote sources.

How many chunks should I put in a RAG prompt?

Commonly 3 to 10, depending on chunk size and context window. Test to find the best balance.

Is RAG better than fine-tuning?

For adding knowledge that changes often, RAG is usually easier and cheaper. Fine-tuning is better for teaching style or format.