RAG How It Works

To understand how RAG works, split it into two pipelines: an indexing pipeline that runs ahead of time to prepare your documents, and a query pipeline that runs every time a user asks a question.

The Indexing Pipeline

• Load: read PDFs, web pages, database rows or other sources into text.
• Clean: remove headers, footers, boilerplate and broken characters.
• Chunk: split long text into passages of a few hundred tokens.
• Embed: convert each chunk into a vector with an embedding model.
• Store: save text, vector and metadata (source, page, date) in a vector database.

The Query Pipeline

• Embed the user question with the same embedding model.
• Search the vector database for the top k most similar chunks.
• Optionally rerank or filter the results.
• Build a prompt containing instructions, the chunks and the question.
• Generate the answer with an LLM and return it with citations.

The Pipelines as Code Outline

Here is the shape of a RAG system as Python function stubs. Later lessons fill in each function.

def index(files):
    for text, meta in load(files):
        for chunk in split(text):
            vector_db.add(embed(chunk), chunk, meta)

def answer(question):
    hits = vector_db.search(embed(question), k=4)
    prompt = build_prompt(question, hits)
    return llm(prompt), [h.meta for h in hits]

(outline only: functions are implemented in later lessons)

Notice that embed is used in both pipelines. Using the same model for documents and questions is essential, because only vectors from the same model live in the same space.

Why Not Paste Everything into the Prompt?

Modern models have large context windows, so for a few short documents you can skip retrieval. But stuffing hundreds of pages is slow, expensive, hits limits, and models are worse at finding facts buried in very long contexts. Retrieval sends only what matters.

Where RAG Systems Fail

• Bad parsing: tables or scanned pages turn into garbage text.
• Bad chunking: the answer is split across chunks or drowned in noise.
• Retrieval misses: the question uses different words than the document.
• Prompt issues: the model ignores context or answers from memory.
• Stale index: documents changed but were not re-indexed.

Most of this tutorial is about fixing these failure points, one stage at a time.

Frequently asked questions

What are the main components of a RAG system?

A document loader, a chunker, an embedding model, a vector database (retriever), a prompt template and an LLM.

How often should I re-index documents?

Whenever they change. Many systems re-index changed files incrementally on a schedule or on upload.

Is retrieval always vector search?

No. Keyword search (BM25), SQL queries, web search or knowledge graphs can all be the retrieval step. Vector search is simply the most common.