Module 1 of 2 · Lesson 2 of 3
Lesson · Module 1 · GenAI System Design
Grounding answers in your own data
Decide when a model needs retrieved context, how that context is chosen and fitted into a prompt, and how stale or missing data changes the answer.
Builds on
A language model answers from two places: what its weights picked up during training, and what is in the request. Ask the help assistant from the previous lesson how many days of parental leave the company offers, and the model has no way to know unless the policy text is in front of it. Without that text it will usually produce a fluent, plausible answer anyway. Grounding is the practice of putting the relevant source material into the request and asking for an answer based on it, so the answer can be checked against something.
This lesson covers when a design needs retrieval to do that, how the supporting context is chosen and fitted into the token budget, and what goes wrong when the underlying data is stale, missing or badly split. The goal is that you can decide whether to add retrieval, describe the pipeline in an interview, and name its failure modes without being asked.
Retrieval is one way to ground, not the default#
Retrieval-augmented generation, or RAG, was introduced as a way to combine a model's parametric memory (its weights) with a non-parametric memory that is looked up at query time. In the original paper the second memory was a dense vector index of Wikipedia reached through a neural retriever (Lewis and others, 2020). The same authors framed provenance and updating knowledge as open problems for language models in general, which is a fair description of why teams reach for it. The term now covers a broader family of designs, so in a discussion it helps to say which parts you mean.
Before adding a retrieval pipeline, work through the cheaper options in order:
| Situation | Better starting point | Why |
|---|---|---|
| The whole source fits comfortably in the token budget and changes rarely | Put it in the prompt | No index, no retrieval errors. Cost is the input tokens on every request, which may be reducible by reusing a repeated prefix. |
| The answer is a precise stored fact, such as an order status or a leave balance | Query the system of record directly and give the result to the model | Embedding similarity is the wrong tool for an exact lookup, and a retrieved near-match can be confidently wrong. |
| The corpus is large, changes, and answers need to point to a source | Retrieval | The model sees only the few passages relevant to the question, and each can be traced back. |
| The model should adopt a style or a skill rather than know a fact | Not a retrieval problem | Changing weights is a different tool, and the choice between the approaches is a topic for later in the Track. |
Many real assistants combine two of these: a direct lookup for the employee's own data and retrieval for policy documents.
The pipeline, and why each step exists#
Two parts of the diagram are easy to leave out of a whiteboard sketch: the filter runs before ranking, and a question can end without a model answer. The filter is explained under finding candidates, and the second path under missing data in what goes wrong.
Splitting documents#
A model can only use what fits in the request, and an embedding represents a whole chunk with a single vector. Both facts push on chunk size from opposite sides. Chunks that are too small lose the context that gives a sentence its meaning, such as the "except for contractors" that sits two paragraphs after the rule it modifies. Chunks that are too large blur many topics into one vector, so retrieval becomes less precise, and each retrieved chunk spends more of the token budget. Overlapping neighbouring chunks softens the boundary problem at the price of storing and sometimes retrieving the same text twice. Splitting along the document's own structure, such as headings and list items, usually does better than splitting at a fixed length, and keeping the heading path with each chunk helps both retrieval and citation. Chunking long inputs for models covers the boundary rules, overlap and attached context in more detail.
Finding candidates#
An embedding model maps text to vectors so that text with similar meaning lands close together, and retrieval returns the chunks nearest to the question. Pure vector search handles paraphrase well, but it can miss queries that depend on an exact string, such as an error code, a policy number or a product name, where keyword matching often does better. Combining keyword and vector scores is a common response. A reranker is a second, more expensive model that rescores the top candidates; it can improve precision but adds latency and cost to every request, so it's a lever to justify with measurements and not a default.
Metadata matters as much as the vectors. Document date, owner, version and permission scope let the system filter before ranking: only current policies, only documents this user may see.
Fitting the context into the budget#
Retrieved text competes with everything else for the token budget from the previous lesson. A deliberately small example keeps the arithmetic visible:
budget 8,000 tokens
- instructions 600
- user question 100
- reserved for the answer 500
= left for retrieved context 6,800
chunk size 400 tokens -> at most 17 chunks fit
The 17 counts chunk text only. Per-chunk metadata such as identifiers and heading paths, and the separators and formatting that wrap each passage in the prompt, also use tokens, so the real number is lower. The window is a ceiling, not a target. Passing 17 chunks because they fit raises cost and latency, and the model has to find the relevant lines among irrelevant ones. Position also matters: the 2023 results on long inputs described in the previous lesson found accuracy highest at the start and end of the input and lower in the middle (Liu and others, 2023). A reasonable heuristic, to confirm by testing on your own data, is to retrieve fewer and better passages and to place the strongest evidence at the edges of the context.
The instructions in the prompt do part of the grounding work: answer only from the provided passages, say so when they don't contain the answer, and cite the identifier of each passage used. Those instructions reduce unsupported answers without eliminating them. A model can still misread a passage or blend it with what it learned in training, which is why the output has to be evaluated and not assumed faithful.
What goes wrong#
Each failure below produces an answer that sounds fine, so each needs a deliberate check.
- Retrieval misses. The answer exists in the corpus but the right chunk isn't among those returned, because of a vocabulary mismatch, a poor split or a filter that was too strict. The model then either admits ignorance or improvises. Diagnose retrieval and generation separately: first ask whether the needed passage was retrieved at all.
- Stale data. The index trails its sources. A superseded policy keeps getting retrieved because its replacement hasn't been re-embedded, or a deleted document remains searchable. The freshness the product needs decides the ingestion design. A nightly batch is simple and acceptable if a day of lag is harmless. If a change must reach answers within minutes, the design needs incremental updates, and deletions have to propagate as deliberately as additions.
- Missing data. The question is simply out of scope for the corpus. An ordinary top-k search without a relevance threshold returns up to k eligible candidates, meaning those that pass any filters, when that many are available, whether or not any of them is relevant. A question the corpus can't answer therefore tends to come back with candidates anyway, and the model may try to use them. Filters and relevance thresholds can reduce the number returned or leave no candidates, but nearest-neighbour ranking alone doesn't establish that a returned chunk is relevant. Similarity scores aren't a reliable relevance judgment on their own, so set any threshold from examples that include unanswerable questions. Give the system an explicit "nothing relevant found" path, decide what the assistant says in that case, and test that behavior.
- Conflicting or badly split sources. Two versions of a policy, or one rule separated from its exception, produce confident answers that are half right. Version and effective-date metadata let retrieval prefer the current one, and keeping each rule with its exceptions in the same chunk avoids the second problem.
- Access leaks. Permission checks must happen at retrieval time, before the prompt is assembled. Once a restricted passage is in the context, the application can't count on the model to withhold it.
- Untrusted text in the prompt. Retrieved documents are data, but a document can contain text that reads like an instruction. Treat retrieved content as untrusted input, keep it clearly separated from the instructions, and plan for it in the failure handling covered later in the Track. No prompt wording is a complete defense by itself.
Retrieval quality is often the real limit on answer quality. A stronger model can't reliably compensate for incorrect or missing retrieved evidence. If answers are bad, look at what was retrieved before changing the model or the prompt.
Using this in a design discussion#
A short sequence usually works: establish what the data is, how large it is, how often it changes, who may see which parts, and whether answers must cite sources. Then pick the simplest arrangement that satisfies those constraints, and name the first change that would push you toward more machinery, such as a corpus that outgrows the prompt or a freshness requirement that outgrows a nightly rebuild.
Sample wording for a candidate: "I'd check first whether the policy set fits in the prompt, since that removes retrieval errors entirely. If it doesn't, I'd retrieve with metadata filters for current versions and the user's permissions, measure whether the right passage is in the top results separately from whether the answer is faithful, and refuse when nothing relevant comes back."
Retrieval and generation fail in different ways, so they're measured in different ways. Evaluating output you can't unit-test turns that into a plan: what to measure before building, how judges help and mislead, and what to watch after launch.
Related Knowledge
Practice
Problem
Design a live incident summary that an on-call engineer can trustDesign a system that condenses chat, alerts and deploy events from a live incident into a handoff summary, and decide how to tell whether the summary can be trusted.
MediumBuilds on this lesson · Also applies Chunking long inputs for models