LangChain Text Splitters

LangChain text splitters break long documents into smaller chunks. Chunks are needed because embedding models and context windows have size limits, and because small focused chunks give more precise search results than whole files.

RecursiveCharacterTextSplitter

This is the recommended default. It tries to split on paragraphs first, then lines, then sentences and words, so chunks stay as natural as possible.

from langchain_text_splitters import RecursiveCharacterTextSplitter

text = """LangChain helps you build LLM apps.

It has loaders, splitters and vector stores. Splitters cut long text into chunks.

Chunks are embedded and stored for search."""

splitter = RecursiveCharacterTextSplitter(chunk_size=60, chunk_overlap=10)
for chunk in splitter.split_text(text):
    print(repr(chunk))

'LangChain helps you build LLM apps.'
'It has loaders, splitters and vector stores. Splitters'
'Splitters cut long text into chunks.'
'Chunks are embedded and stored for search.'

chunk_size is the maximum length of each chunk (in characters by default). chunk_overlap repeats a little text between neighbouring chunks so ideas cut at a boundary are not lost. Exact splits depend on the separators found.

Splitting Documents and Keeping Metadata

from langchain_community.document_loaders import PyPDFLoader

pages = PyPDFLoader("handbook.pdf").load()
splitter = RecursiveCharacterTextSplitter(chunk_size=1000, chunk_overlap=150, add_start_index=True)
chunks = splitter.split_documents(pages)
print(len(pages), "pages ->", len(chunks), "chunks")
print(chunks[0].metadata)

Sample output:
24 pages -> 71 chunks
{'source': 'handbook.pdf', 'page': 0, 'start_index': 0}

Splitting by Tokens

Models count tokens, not characters. from_tiktoken_encoder measures chunk size in tokens (pip install tiktoken):

splitter = RecursiveCharacterTextSplitter.from_tiktoken_encoder(chunk_size=300, chunk_overlap=40)
print(len(splitter.split_text(text * 50)))

Sample output:
3

Markdown and Code Splitters

from langchain_text_splitters import MarkdownHeaderTextSplitter

md = "# Setup\nInstall Python.\n## Windows\nUse the installer.\n# Usage\nRun main.py."
splitter = MarkdownHeaderTextSplitter([("#", "h1"), ("##", "h2")])
for d in splitter.split_text(md):
    print(d.metadata, "|", d.page_content)

{'h1': 'Setup'} | Install Python.
{'h1': 'Setup', 'h2': 'Windows'} | Use the installer.
{'h1': 'Usage'} | Run main.py.

Choosing Chunk Size

• Start with 500 to 1000 characters and 10-20% overlap.
• Smaller chunks give precise matches but less context per chunk.
• Larger chunks keep context but can dilute the embedding.
• Use RecursiveCharacterTextSplitter.from_language(...) for source code.
• Always test with real questions; the RAG tutorial covers evaluation.

Frequently asked questions

What is a good chunk size for RAG?

A common starting point is 500 to 1000 characters (roughly 100 to 250 tokens) with 10-20% overlap, then tune based on retrieval quality.

What does chunk_overlap do?

It repeats the end of one chunk at the start of the next, so a sentence split across a boundary still appears whole in at least one chunk.

Which package contains LangChain text splitters?

The langchain-text-splitters package. Import from langchain_text_splitters.