A language model answers from patterns compressed into its parameters. Retrieval-augmented generation, or RAG, adds a second kind of memory: an external collection searched when a question arrives. The model receives selected passages with the prompt and uses them to construct an answer. That seemingly modest architectural change makes knowledge easier to update, restrict, and inspect without retraining the entire model.
The 2020 paper that popularized the term combined a sequence-to-sequence generator with a dense index of Wikipedia. Its motivation remains current: “providing provenance for their decisions and updating their world knowledge remain open research problems.” On three open-domain question-answering tasks, the authors reported state-of-the-art results at the time and found their models generated more factual, specific, and diverse language than a parametric-only baseline. The NeurIPS paper established the pattern, not a guarantee that every later RAG system is grounded.
From documents to answer-time context
A typical pipeline first collects approved documents, removes duplicates, and splits them into chunks. An embedding model converts each chunk into a numerical representation, which is stored in a searchable index. When a user asks a question, the system embeds the query, retrieves similar chunks, often combines semantic search with exact-term search, reranks candidates, and sends a small set to the generator.
Each step encodes a policy choice. Chunk size determines whether a definition stays with its qualifications. Metadata filters decide which date, jurisdiction, department, or user's files are eligible. The number of retrieved passages trades coverage against cost and distraction. The final prompt determines whether the model should abstain when evidence is missing and how citations must be formatted.
The approach is valuable because the corpus can change independently of the model. A benefits assistant can index a new policy on the day it takes effect. A research tool can limit answers to a curated library. An enterprise can enforce document permissions at retrieval time rather than train confidential material into shared weights. NIST's definition captures the essential structure: a generative model paired with a separate information-retrieval system or knowledge base.
Better retrieval means better chances, not proof
Retrieval is often the limiting component. Semantic embeddings can match related concepts, while lexical methods such as BM25 are better at exact identifiers, names, and error codes. Hybrid systems combine them and use a reranker to score a smaller candidate set.
In one vendor experiment across code, fiction, scientific papers, and other domains, Anthropic found that adding short document context to chunks reduced its top-20 retrieval failure rate from 5.7% to 2.9%, a 49% reduction; adding reranking brought it to 1.9%, a 67% reduction. The Contextual Retrieval report discloses the metric and setup, so the figures should be treated as configuration-specific rather than universal.
More context is not always better. The Lost in the Middle study found that models often performed best when relevant information appeared at the beginning or end of a long input and worse when it appeared in the middle. In an open-domain experiment, expanding retrieval from 20 to 50 documents improved performance only about 1.5 percentage points for GPT-3.5-Turbo and about one point for Claude 1.3, even as retriever recall continued to rise. The peer-reviewed study shows why dumping an entire archive into a prompt is not the same as using evidence well.
Citations create an auditable surface
RAG can expose the passages used for an answer, which allows a reader or an automated evaluator to ask three separate questions. Did the system retrieve the right source? Does the cited passage support the nearby claim? Did the answer preserve qualifications and avoid unsupported additions?
Those checks are stronger than accepting fluent text, but a citation can still be decorative. A model may cite a relevant document that does not entail its sentence, combine incompatible dates, or choose a secondary summary over the controlling source. Evaluation should therefore score retrieval recall, answer correctness, citation precision, entailment, completeness, latency, and abstention—not a single generic accuracy number.
NIST's National Cybersecurity Center of Excellence documented these tradeoffs in a 2025 prototype that searched NIST cybersecurity publications. Its technical report identifies prompt injection, hallucination, data exposure, and unauthorized access as design threats, and describes local deployment, access controls, and validation filters. An indexed document is untrusted input as well as evidence: hidden instructions in a retrieved page should not override the system's rules.
Evidence becomes a system property
RAG works best when ownership is explicit. Someone must curate sources, preserve versions, attach dates and permissions, test representative questions, and investigate failures. The generator should distinguish quoted evidence from inference and make uncertainty visible. For high-stakes decisions, retrieval can assist a qualified reviewer rather than replace one.
The field is developing shared tests. NIST's 2025 TREC RAG track separated passage retrieval, augmented generation, full RAG, and relevance-judgment tasks—an acknowledgment that a strong answer depends on several measurable components.
RAG does not give a model truth. It gives the system a controllable path to evidence. That path can be updated and audited, but it can also retrieve the wrong material or use the right material badly. Its lasting contribution is to make factual quality partly an engineering and governance problem outside the model—where sources, permissions, citations, and tests can be inspected.