What RAG really is — and when not to use it

Retrieval-augmented generation gets described as if it were a single feature you switch on. It isn't. RAG is a small pipeline of deliberate steps, each with its own failure modes, and the quality of the final answer is bounded by the weakest one. I've built and operated retrieval systems over large document sets, and the thing I keep coming back to is that most "RAG problems" are actually retrieval problems the generation step can't fix.

This post walks the whole thing end to end, then spends real time on the part people skip: when you should not reach for RAG at all.

The pipeline, step by step

At its core, RAG answers a question by first fetching relevant text and then asking a language model to answer using only that text. The model supplies fluency and reasoning; your retrieval layer supplies the facts. Here is the shape of it.

1. Chunking

You can't retrieve a whole PDF, so documents get split into chunks — typically a few hundred tokens with some overlap. This sounds trivial and is quietly one of the highest-leverage decisions in the system. Chunk too large and a single embedding has to represent several unrelated ideas, so it matches nothing well. Chunk too small and you shred the context a sentence needs to make sense. Respecting document structure — headings, tables, list boundaries — usually beats a naive fixed-size window.

2. Embeddings

Each chunk is passed through an embedding model that maps text to a vector — a few hundred to a few thousand floating-point numbers — where semantic similarity becomes geometric closeness. "How do I reset my password" and "forgot login credentials" land near each other even with no shared words. You embed every chunk once at ingest time and store the vectors.

3. Vector search (top-k)

At query time you embed the question with the same model and find the nearest chunk vectors, usually via an approximate nearest neighbor index (HNSW, IVF) because exact search over millions of vectors is too slow. You pull the top k — say 20 to 50 candidates. This step favors recall: get the right chunk somewhere in the set, and let a later step sort out precision.

4. Reranking

Embedding similarity is fast but coarse. A reranker — often a cross-encoder that reads the query and each candidate together — scores relevance far more accurately, then you keep the top 3 to 8. This is where many systems find their biggest quality jump, because it fixes the "retrieved but buried" problem where the right chunk was rank 18 and never made it into the prompt.

5. Grounding the prompt

The surviving chunks get stitched into the prompt with an instruction to answer only from the provided context and to say so when the answer isn't there. This is the guardrail against hallucination: you're not asking the model what it knows, you're asking it to read.

System: Answer using ONLY the context below.
If the context is insufficient, say so. Cite sources as [n].

Context:
[1] {chunk_1}  (doc: billing.md, p.4)
[2] {chunk_2}  (doc: refunds.md, p.1)

Question: {user_question}

6. Citations

Because each chunk carries its source metadata, the model can tag claims with references and you can render links back to the original document. Citations aren't decoration — they're how a user (or an auditor) verifies the answer, and they turn "trust me" into "check for yourself."

Lessons from running RAG at scale

Once a corpus grows into the millions of pages, a few things stop being optional. These are general lessons, not any one product's secret sauce:

• Latency is a budget you spend across stages. Embedding the query, ANN search, reranking, and generation each add milliseconds. Reranking 50 candidates with a heavy cross-encoder can cost more than the generation itself, so you tune k and the reranker size against a p95 target rather than chasing a theoretical best.
• Prefer deterministic steps over extra LLM calls.It is tempting to add an LLM "query rewriter" or an LLM "relevance filter" at every stage. Each one adds latency, cost, and a new way to be non-deterministic. Metadata filters, regex, and a good reranker often do the same job faster and more predictably.
• Cost scales with tokens, not with cleverness. Stuffing 15 chunks into context feels safe but inflates every single call. Fewer, better-ranked chunks usually beat more chunks, on both quality and bill.
• Evaluation has to be continuous. A change to chunking or the embedding model can silently regress retrieval. A held-out set of question/answer pairs with retrieval hit-rate metrics catches this; eyeballing does not.

When RAG is the wrong tool

RAG is a means, not a goal. Plenty of teams bolt a vector database onto a problem that never needed one. A few honest cases where I'd push back:

• The corpus is small. If your entire knowledge base is a few dozen pages, a long-context model can simply read all of it in the prompt. You skip the whole ingest-index-retrieve apparatus and its failure modes. Retrieval earns its complexity only when the corpus is too big to fit or too big to be cheap.
• The question is a structured lookup."What was order #4471's total?" is a database query, not a semantic search. Fuzzy vector matching over structured data is strictly worse than SQL — slower, less accurate, and unable to aggregate. If the answer lives in rows and columns, query the rows and columns.
• You need behavior, not facts. RAG injects knowledge at inference time. It does not teach a model a new format, tone, or task convention. If you want the model to consistently output a specific structure or adopt a domain style, fine-tuning (or few-shot examples) is the right lever, not retrieval.
• The knowledge is already in the model.For general, stable knowledge that the base model handles well, a retrieval round trip adds latency and cost for no accuracy gain. Reserve RAG for what the model can't or shouldn't memorize: your private, fast-changing, or citation-requiring data.

The useful mental model is that RAG buys you freshness, privacy, and verifiabilityat the cost of a pipeline you now have to maintain and evaluate. When those three things matter, it's the best tool we have. When they don't, a long-context prompt, a SQL query, or a fine-tune will serve you better — and with far fewer moving parts.