RAG Production Tips

Moving RAG to production is about reliability, security and cost, not just accuracy. These RAG production tips come from the problems that appear once real users and real documents arrive.

Indexing Pipeline

• Re-index incrementally: hash each file and only process changed ones.
• Use deterministic chunk IDs so updates replace old chunks.
• Delete chunks when source documents are deleted.
• Store embedding model name and chunking settings with the index.
• Invest in parsing: tables, headings and OCR quality drive answer quality.

Incremental Indexing Example

import hashlib, json, pathlib

state_file = pathlib.Path("index_state.json")
state = json.loads(state_file.read_text()) if state_file.exists() else {}

def needs_reindex(path):
    digest = hashlib.sha256(pathlib.Path(path).read_bytes()).hexdigest()
    changed = state.get(path) != digest
    state[path] = digest
    return changed

for f in ["faq.txt", "handbook.txt"]:
    print(f, "reindex" if needs_reindex(f) else "skip")
state_file.write_text(json.dumps(state))

Sample output (first run):
faq.txt reindex
handbook.txt reindex

Security and Access Control

Store permissions (team, user, tenant) in chunk metadata and always filter retrieval by the current user, so the model never sees documents the user cannot open.

Retrieved documents can contain prompt injection, text like "ignore previous instructions". Treat retrieved content as untrusted data: keep it clearly separated in the prompt, never give a RAG model dangerous tools without checks, and filter outputs.

Latency and Cost

• Cache embeddings for repeated queries and cache full answers for common questions.
• Stream the answer so users see text immediately.
• Keep k small after reranking; fewer tokens means faster, cheaper answers.
• Use a small model for query rewriting and a larger one only for final answers.
• Run Ollama or vLLM on a GPU server for multi-user local deployments.

Monitoring

Log every question, the retrieved chunk IDs, scores, the final answer, latency and user feedback (thumbs up/down). Review low-rated and "I don't know" answers weekly; they show missing documents and retrieval gaps. Feed good examples back into your evaluation set.

Graceful Failure

When retrieval scores are low, say so and suggest rephrasing rather than guessing. Showing sources, a confidence hint and a way to contact a human keeps trust high when the system is unsure.

Frequently asked questions

How do I keep a RAG index up to date?

Track file hashes or modification times, re-embed only changed documents, delete removed ones, and run the job on a schedule or on upload events.

Is RAG vulnerable to prompt injection?

Yes. Malicious text inside documents can try to steer the model. Treat retrieved text as data, limit tool permissions and validate outputs.

How do I make RAG faster?

Stream responses, cache, use ANN indexes, keep prompts short with reranking, and run models on GPUs.