RAG Chunking Strategies

RAG chunking strategies decide how your documents are cut into pieces before embedding. Chunking quietly controls retrieval quality: if a chunk does not contain the whole answer, or contains too much unrelated text, even the best model will struggle.

Fixed-Size Chunking with Overlap

The simplest method cuts every N characters with some overlap. It is easy but can cut sentences in half.

def fixed_chunks(text, size=200, overlap=40):
    step = size - overlap
    return [text[i:i + size] for i in range(0, len(text), step)]

text = "RAG retrieves context. " * 20
chunks = fixed_chunks(text)
print(len(text), "chars ->", len(chunks), "chunks")
print(repr(chunks[1][:30]))

460 chars -> 3 chunks
' RAG retrieves context. RAG re'

Paragraph-Aware (Recursive) Chunking

A better default packs whole paragraphs into chunks and only splits further when a paragraph is too long. This mirrors how LangChain's RecursiveCharacterTextSplitter works.

def paragraph_chunks(text, max_chars=500):
    chunks, current = [], ""
    for para in text.split("\n\n"):
        if len(current) + len(para) + 2 <= max_chars:
            current = (current + "\n\n" + para).strip()
        else:
            if current:
                chunks.append(current)
            current = para[:max_chars]   # very long paragraphs: split further in real code
    if current:
        chunks.append(current)
    return chunks

doc = "Intro paragraph.\n\n" + "Policy details. " * 20 + "\n\nContact us at help@example.com."
for c in paragraph_chunks(doc, 200):
    print(len(c), c[:40])

16 Intro paragraph.
200 Policy details. Policy details. Policy d
31 Contact us at help@example.com.

Other Chunking Strategies

• Sentence-based: group N sentences per chunk; good for prose.
• Structure-aware: split on Markdown/HTML headings, keeping the heading path in metadata.
• Semantic: start a new chunk when embedding similarity between sentences drops.
• Document-specific: one chunk per FAQ entry, product, or database row.
• Parent-child: search small chunks, but send their larger parent section to the LLM.

Choosing Chunk Size and Overlap

There is no universal best size. A common starting point is 300 to 800 tokens-worth of text (roughly 1000 to 3000 characters) for long documents, or smaller (100 to 300 tokens) for FAQ-like content, with 10-20% overlap. Small chunks give precise matches but lose context; big chunks keep context but blur the embedding and cost more tokens in the prompt.

The only reliable way to choose is to test with real questions and measure retrieval, as shown in the evaluation lesson.

Add Context to Every Chunk

def with_context(chunk, title, section):
    return f"Document: {title}\nSection: {section}\n\n{chunk}"

print(with_context("Employees get 20 days of leave.", "HR Handbook 2026", "Leave Policy"))

Document: HR Handbook 2026
Section: Leave Policy

Employees get 20 days of leave.

Prepending the document title and section ("contextual chunking") helps a chunk that says "it" or "this policy" still match the right questions.

Frequently asked questions

What is the best chunk size for RAG?

Start around 500 to 1000 characters with 10-20% overlap, then tune by measuring retrieval on real questions. Content type matters more than any rule.

Is semantic chunking better than fixed-size?

Sometimes, but it is slower and benchmarks show mixed results. Recursive, structure-aware chunking is a strong, cheap default.

Should chunks overlap?

Usually a small overlap helps so facts near boundaries are not lost. Too much overlap creates near-duplicate results.