RAG: Retrieval-Augmented Generation, Pipelines, Skip Rules

Retrieval-augmented generation (RAG) is a system that pairs a frozen language model's parametric memory with a retrievable non-parametric corpus. Lewis et al. introduced it at NeurIPS 2020 using DPR plus BART.

ChatGPT is a product; RAG is a pattern. Long context did not delete retrieval: Anthropic skips RAG only under about 200,000 tokens.

Menlo Ventures put RAG in 51% of enterprise genAI design patterns in 2024, against 9% fine-tuning and 12% agentic.

Key Takeaways

• RAG is a system around a model. You swap the corpus without touching weights, which is the original Lewis design.
• ChatGPT is not "a RAG." Some OpenAI features retrieve, including File Search on the Responses API.
• Production retrieval is hybrid BM25 plus dense plus a reranker. Dense-only nearest neighbor is the demo.
• Packing order is a generation failure. Liu et al. show answers degrade in the middle of long context.
• Skip RAG when the corpus fits in the prompt, the question is a SQL aggregation, or you still have a chunking and freshness problem. Do not add an agent to paper over a bad index.

What Is RAG?

RAG is retrieval-augmented generation for large language models: you retrieve passages at inference, then generate an answer conditioned on those passages. Wikipedia restates the public one-liner as a parametric language model plus non-parametric external memory. The definition still belongs to the 2020 paper.

Lewis, Perez, Piktus, Petroni, Karpukhin, Goyal, Küttler, Mike Lewis, Yih, Rocktäschel, Riedel, and Kiela built two memories. Parametric memory was a pre-trained seq2seq (BART). Non-parametric memory was a Wikipedia dense index accessed with DPR.

The original variants were RAG-Sequence and RAG-Token. Meta's 2020 writeup called the trick late fusion: swap the corpus, keep the weights frozen, change the answers. Hugging Face still ships facebook/rag-sequence-nq.

Patrick Lewis is now Director of Agentic AI at Cohere. Douwe Kiela, a co-author, later ran Contextual AI and is not the author of Anthropic's Contextual Retrieval method.

RAG is not a model. It sits on top of one.

Kian Katanforoosh puts it in Stanford CS230 (Stanford Online, 47:02):

"You don't touch the model with RAGs. It's really a technique that is put on top of the model."

Is ChatGPT a RAG? No. ChatGPT is a product, and RAG is a pattern.

File Search on the Responses API is hosted retrieval (semantic plus keyword over vector_stores). Assistants File Search is deprecated with the Assistants API; the sunset was 26 August 2026. Do not design against Assistants.

Why is RAG outdated? It is not, unless you think retrieval is dead.

Long context and agents changed the routing decision. They did not delete keyword search, metadata filters, or a corpus that updates after training.

Why RAG Matters in 2026

You use RAG when the facts change, the answer must be cited, or you refuse to fine-tune every time a policy PDF lands.

The 2026 papers refuse the 2020 dense-supremacy story as a production default. Akarsu, Karaman, and Mierbach ran 23,088 financial text-and-table queries: BM25 beat standalone dense models; hybrid plus neural rerank reached Recall@5 0.816 and MRR@3 0.605.

Wang et al. found BM25 overtaking a file-system agent around 10 million corpus tokens and leading every larger shared tier. Dense retrieval stayed efficient but less accurate.

Agents made the consumer of RAG bigger than a chat box. Palvi Nargun said it in AWS re:Invent 2025 NTA403: "The concept of user is also changed. It's not only just a human who's going to be querying the RAG, but it is also autonomous systems." (AWS Events, 4:01)

Agents querying private corpora is why this sits next to MCP and LLM observability, not a vector-database bake-off. Index the chunks in the store you already run. The architecture question is how you retrieve, pack, evaluate, and refuse to retrieve.

How RAG Works: The Retrieval Pipeline

The boring baseline is four steps: prepare and chunk documents, index vectors, retrieve, stuff the prompt. Production adds parse quality, hybrid search, rerank, and packing order.

Treat the pipeline as ingest → chunk → embed → hybrid retrieve → rerank → assemble → generate.

Ingest and Parse

Most demos start at embeddings. Most outages start at PDFs.

LlamaIndex's 2026 warning is that pipelines parse every page before anyone asks. For an agent in an ad-hoc data room that is slow, expensive, and mostly unread.

The better pattern is just-in-time OCR in two passes: cheap page hits, then expand the section you actually need. If the job is invoices, receipts, or ACORD forms, extract fields into rows. Do not pretend a table is a paragraph.

Chunking

Fixed token splits (100, 256, 512) are still the common starting recipe. Larger chunks add context and noise. Tiny chunks lose coherency.

Start near 512 tokens because that is what Wang et al. measured, not because a vendor remix said so. On lyft_2021, chunk 512 scored faithfulness 97.59 and relevancy 97.41. Chunk 256 was 97.22 / 97.78, and chunk 2048 dropped to 80.37 / 91.11.

The retrieval unit is a design choice: document, passage, sentence, or proposition. RAPTOR (Sarthi et al., ICLR 2024) builds a recursive abstractive tree of ~100-token segments for multi-hop and whole-document questions.

Use it when the question needs themes across a file. Do not make it the default index.

On r/LocalLLaMA, the recurring production complaint is that off-the-shelf chunkers ignore your data. Traceability beats a framework default.

Hybrid Retrieve and Rerank

Dense-only nearest neighbor is how you demo RAG. It is not how you ship it.

Anthropic Contextual Retrieval (19 September 2024) prepends chunk-specific context before embedding. Failure is 1 − recall@20. Anthropic reports that contextual embeddings cut top-20-chunk retrieval failure 35% (5.7% → 3.7%).

Adding contextual BM25 reached 49% (5.7% → 2.9%). Adding a reranker reached 67% (5.7% → 1.9%). Chunks in that post are usually a few hundred tokens.

Ubuntu's 2026 writeup splits the jobs cleanly: embeddings for geometry, BM25 for exact tokens (SKUs, error codes), Reciprocal Rank Fusion to merge lists, a cross-encoder to rerank. Pinecone said the literal-string failure in public:

"Semantic search has never been good at literal strings. Query a string literal like a part number and all the similar part numbers score about as high as the one you actually wanted, and your agent answers from whatever lands on top, whether the data is correct or not." (Pinecone on LinkedIn, Sep 2026)

On r/LangChain, the 2026 shift is the same stack: BM25 plus vectors plus a cross-encoder or LLM reranker, plus query rewriting. Ranking, not raw recall, is the bottleneck you actually feel.

Cohere Rerank is one named commercial rerank stage. Treat it as a stage, not as the architecture.

OpenAI File Search already runs semantic and keyword search over vector_stores.

Assemble and Generate

Retrieval can succeed and generation can still fail.

Liu et al. (Lost in the Middle, TACL 2024) showed performance peaking when evidence sits at the beginning or the end of the context. The middle degrades, including on long-context models.

Dumping an unordered top-20 is an assembly bug. Adding a tenth chunk to raise recall can bury the answer-bearing chunk.

Citation correctness is also not faithfulness. A model can cite a passage it did not use. If you evaluate only "did we retrieve it," you will ship fluent, well-cited wrong answers.

Parent-document and multi-vector retrievers embed small chunks, summaries, or hypothetical questions, then expand to the parent section at generation. Two-pass section retrieval gets GraphRAG-shaped coverage without a graph database.

Naive RAG vs Agentic Control Loops

Do not learn architecture from a 14-types listicle. Use Gao et al. 2023: Naive RAG is retrieve-then-read (index → top-K → stuff prompt).

Advanced RAG adds pre-retrieval and post-retrieval. Modular RAG is replaceable components.

Azure's architecture guide draws the control-flow line. Standard RAG is a fixed-sequence orchestrator: you decided to search at design time. Agentic RAG treats retrieval as a tool the agent invokes, evaluates, and iterates.

Google Research (5 June 2026) is blunt about the single-step miss: "Project X server specs" may put the ID in one corpus and the specs in another. Anthropic's agent guide separates workflows from agents. Context engineering treats RAG as one way to curate a finite window, not the only way.

Singh et al. survey agentic RAG as iterative routing and refinement. Their constraint still matters: retrieval quality remains the primary bottleneck. Agentic reasoning does not repair a bad index.

In April 2026, u/andrew_kirfman put the default in one line: start simple, add complexity only when a simpler loop is actually insufficient.

"All this looks cool on paper and then you ask about logging or audit and the architecture falls apart. Haven't seen a single diagram showing Auth, log trace, observability, telemetry, and risk mitigators." (u/gob_magic in r/LangChain, Apr 2026)

If failures come from bad chunking or stale files, fix those before you pay for a reasoning loop. Simple single-source lookups do not need one. MCP is adjacent here: tool access is not a private-corpus index.

Advanced Patterns as Failure-Mode Matches

Match the pattern to the failure. GraphRAG and agentic RAG are failure-mode matches, not the default architecture.

CRAG on SelfRAG-LLaMA2-7b reported +7.0% PopQA and +14.9 FactScore Biography versus RAG. RAPTOR plus GPT-4 gained about 20% absolute on QuALITY versus the prior best. Those are pattern-match numbers, not a reason to rebuild your index this week.

GraphRAG is the loudest near-variant. Microsoft created the repo on 27 March 2024. The README is in maintenance mode with no new features.

The MSR blog is the 2024 research system, not Microsoft's current SKU. If your questions are joins and aggregations, you probably wanted SQL.

How to Evaluate a RAG System

Glossaries skip evaluation. Tool roundups fill that SERP. You need the split, not another Ragas landing page.

Three surfaces:

• Retrieval quality (did the right chunks land, and in what rank?).
• Faithfulness / groundedness of generation (did the answer use those chunks, and only those chunks?).
• Systems: latency, embedding drift, ACLs, traces. This is LLM observability, not a single score.

RAGAS (Es, James, Espinosa-Anke, Schockaert; EACL 2024) named the common metrics. Faithfulness is (claims supported by retrieved context) / (claims in the response), 0–1.

Context Precision asks whether relevant chunks ranked high (mean precision@k). Sibling metrics include Context Recall, Noise Sensitivity, and Response Relevancy.

Faithfulness is not correctness. An answer can be faithful and wrong, or correct and unfaithful. Platform scores for "helpfulness" without a task definition are how teams adopt eval, get a useless number, and quit.

LangSmith (LangChain) and Langfuse are eval-loop infrastructure, not product reviews. Eval capacity is the scarce resource, not another dashboard.

Yu et al. treat retrieval and generation as separately scored components. Score retriever, generator, and the live system separately, or you will "fix generation" when the index is stale.

Production Trade-Offs That Break Demos

A notebook that answers three questions is not a system.

Barnett et al. (Seven Failure Points, CAIN 2024) is the named FP list you should keep on the wall:

Their takeaway is operational: validation is feasible during operation, and robustness is an engineering problem. The biomedical case used 15,000 BioASQ docs and 1,000 Q-A pairs.

Latency stacks. Embed plus search plus rerank plus pack before a token is generated.

ACLs, prompt injection in retrieved content, and embedding drift are production surfaces, not research footnotes. AWS Bedrock Knowledge Bases is how some enterprises buy a managed pipeline (managed KB vs bring-your-own store). That purchase does not redefine RAG.

Pure vector RAG also fails aggregations. Vector indexes do not sum 100,000 invoices. r/LocalLLaMA's working pattern is the same: receipts become rows in a queryable table.

"For example having receipts, where we extract each individual item and total amount and date… You actually don't want rag but want each invoice turned into one row of a queryable table." (u/Njee_ in r/LocalLLaMA, Sep 2026)

When You Should Not Use RAG

Vendor glossaries skip this list. Use it anyway.

Katanforoosh's latency argument belongs here, from CS230 (Stanford Online, 15:03):

"In theory if we have infinite compute then RAG is useless because you can just read a massive corpus immediately and answer your question. But even in that case, latency might be an issue. Imagine the time it takes for an AI to read all your Drive every single time you ask a question."

RAG vs fine-tuning is complementary, not a binary. Use RAG when knowledge changes, answers must be cited, or you lack labeled fine-tune data.

Use fine-tuning when knowledge is stable and you need behavior. A first-party cost check: Predibase LoRA Land trained 25 Mistral-7B models at under $8 each on average (2024).

Cache-augmented generation (CAG) is the 2025–26 alternative when the working set is small enough to prefill. Tool use via MCP is a different job: calling tools is not retrieving your corpus. Web-grounded agents are not private-corpus RAG.

LangChain, LlamaIndex, and Haystack are how tutorials assemble this. They are not the architecture.

For an app-builder comparison, see AI app builders. Come back here for the retrieval system those builders hide.

Common RAG Mistakes to Avoid

Dense-Only Retrieval on Literals

You embed SKUs, ticket IDs, and error codes, then act surprised when similar strings outrank the exact one. Add BM25 (or another lexical lane) and a reranker before you change embedding models again.

Parse-Everything Ingest

You OCR the entire data room up front. Most pages never get read. Just-in-time parse, then spend extraction budget on the pages the query actually touched.

Unordered Top-K Packing

You retrieved the answer and stuffed it in chunk 11 of 20. Lost in the Middle is the name of that bug. Put the highest-scoring evidence first and last, or retrieve fewer better chunks.

Agentic Layer on a Bad Index

Multi-agent diagrams look like progress. They multiply LLM calls on the same missed document. Singh's survey and the r/LangChain ops complaints agree: retrieval quality first.

Faithfulness as Correctness

A 0.92 faithfulness score means the model stayed inside the retrieved text. If that text is the wrong policy version, you shipped a well-grounded error. Keep retrieval, generation, and systems scores separate.