Building RAG Systems That Actually Work in Production

Retrieval-Augmented Generation has become the go-to architecture for giving LLMs access to custom knowledge. The concept is elegant: instead of fine-tuning a model on your data, you retrieve relevant documents at query time and include them in the prompt. But the gap between a RAG demo and a production RAG system is enormous. Here's what we learned building systems that serve thousands of queries daily.

The Chunking Problem

How you split documents into chunks fundamentally determines retrieval quality. Chunk too large and you dilute relevance with noise. Chunk too small and you lose context. Fixed-size chunking with overlap is the common starting point, but semantic chunking — splitting at natural topic boundaries — dramatically improves results. We found that recursive character splitting with 512-token chunks and 50-token overlap hit the sweet spot for most document types.

from langchain.text_splitter import RecursiveCharacterTextSplitter
 
splitter = RecursiveCharacterTextSplitter(
    chunk_size=512,
    chunk_overlap=50,
    separators=["\n\n", "\n", ". ", " ", ""],
    length_function=lambda t: len(tokenizer.encode(t)),
)
 
chunks = splitter.split_documents(documents)

Embedding Model Selection

Not all embedding models are created equal. OpenAI's text-embedding-3-large is a solid default, but domain-specific models often outperform general-purpose ones. For technical documentation, we saw a 15% improvement in retrieval accuracy by fine-tuning an open-source embedding model on our query-document pairs. The MTEB leaderboard is your friend for benchmarking options.

The quality of your RAG system is 80% retrieval and 20% generation. If you retrieve the wrong context, no amount of prompt engineering will save you.

Hybrid Search: The Secret Weapon

Pure semantic search misses keyword-exact matches. Pure keyword search misses semantic similarity. Hybrid search combines both — using BM25 for sparse retrieval and vector similarity for dense retrieval, then fusing the results with Reciprocal Rank Fusion. In our benchmarks, hybrid search improved answer accuracy by 23% compared to vector-only retrieval.

Evaluation and Monitoring

• Build a golden evaluation dataset of 100+ question-answer pairs
• Track retrieval precision and recall separately from generation quality
• Monitor hallucination rate using automated fact-checking against retrieved context
• Log user feedback and use it to continuously improve chunking and retrieval
• Set up alerts for retrieval failures (queries with low similarity scores)

A RAG system without evaluation is just a demo. Build measurement into the system from day one, and you'll have the data you need to systematically improve every component of the pipeline.