Building a RAG Pipeline That Doesn't Hallucinate

Introduction

Retrieval-augmented generation gets pitched as the cure for hallucinations. Plug a vector store into your prompt, the model reads the right context, and the answers stop being made up.

In practice, naive RAG pipelines hallucinate just as confidently as the model on its own — sometimes worse, because the retrieved context gives the wrong answer a thin coat of plausibility. The patterns below are the ones that actually move the needle.

The Naive Pipeline

The textbook setup is four steps:

1. Chunk your documents.
2. Embed each chunk and store the vectors.
3. At query time, embed the question and pull the top-k chunks.
4. Stuff those chunks into the prompt and ask for an answer.

This works in demos. It falls over the moment your questions stop looking exactly like your chunks.

Where Hallucinations Sneak In

Three failure modes account for almost all of the hallucinations I've debugged in production RAG systems.

Bad retrieval

The top-k chunks don't actually contain the answer. The model either tries to answer from its parametric memory anyway, or confidently summarizes the wrong passage. Better retrieval — hybrid search (vector + BM25), query rewriting, reranking — is usually the single biggest lever.

Weak grounding

The chunks are right, but the prompt doesn't require the model to cite them. Adding "answer only from the provided context; if the answer isn't there, say so" and rewarding citations in your eval set forces the model to stay close to source.

Missing context

The chunk has the answer, but it's stripped of context that made it interpretable — section headings, table structure, parent document. Including a small breadcrumb ( Document → Section → Subsection) with each retrieved chunk often fixes this for almost no extra tokens.

Practical Patterns

The ones I now reach for first, before any embedding-model tuning:

• Hybrid retrieval. Run vector search and keyword search side by side; combine with reciprocal rank fusion.
• Query rewriting. Ask a small model to rephrase the user's question into one or more retrieval queries before searching.
• Reranking. Pull a wider candidate set, then rerank with a cross-encoder to keep the top few that actually match.
• Structured citations. Force the model to emit chunk IDs alongside its answer; reject responses that don't.
• Abstain by default. Make "I don't know" a first-class output. Quality goes up immediately.

Evaluation

None of the patterns above mean anything without an eval set that catches regressions. Write 50–100 questions with known good answers from your corpus, score retrieval and generation separately, and run them on every change.

// retrieval@k: did we pull a chunk that contains the answer?
// faithfulness: does the generated answer only use retrieved text?
// answer correctness: judged by a stronger model, or a human spot-check.

You'll be surprised how often a change that "feels better" in ad-hoc testing makes retrieval@k worse on your eval set.

Closing Thoughts

Good RAG is mostly retrieval engineering with a thin LLM on top. The model is the easy part — the hard part is making sure it gets the right context to ground on, every time.