How RAG Actually Works
A practical walkthrough of retrieval-augmented generation — what to retrieve, how to chunk, and where systems fail.
Retrieval-augmented generation (RAG) is the pattern most teams reach for when they need an LLM to answer questions about their data.
The core loop
- Index your documents into a vector store (or hybrid search index).
- Retrieve the most relevant chunks for a user query.
- Augment the prompt with those chunks.
- Generate an answer grounded in the retrieved context.
Sounds simple. Production is where it gets interesting.
Chunking matters more than the model
If your chunks are too large, retrieval gets noisy. Too small, and you lose context. Start with:
- 400–800 tokens per chunk
- Overlap of ~10–15%
- Metadata: source, section title, date
Then measure. Chunking is an eval problem, not a vibes problem.
Where RAG fails
- Wrong retrieval — the model never sees the answer.
- Stale indexes — docs changed, embeddings didn't.
- Citation theater — the model invents sources that look real.
Fix retrieval first. Better prompts cannot rescue missing context.
A minimal mental model
Treat RAG as a search system with a language model on top. If search is weak, generation will hallucinate confidently. If search is strong, even a mid-tier model looks sharp.