RAG: Retrieval-Augmented Generation

Retrieval-Augmented Generation (RAG) answers questions with a language model plus a search step: the system embeds your question as a vector, finds the k most similar document chunks by cosine similarity, pastes them into the prompt, and instructs the model to answer from that context. The instruction does not guarantee compliance. The model never retrains — it reads.

The honest mental model is an open-book exam with a librarian. The student (the LLM) is brilliant but has never seen your documents. The librarian (retrieval) sprints into the archive and comes back with k pages. Whatever happens next — a correct answer, a confident blunder, an honest "I don't know" — was largely decided by which pages came back. That is the single most useful thing to know about RAG, and it is why this lesson spends as much time on the librarian as on the student.

How RAG Works

Everything happens in two phases, and only one of them runs when you ask a question.

Indexing, before queries, updated when documents change. The documents are split into chunks — a few sentences each — and every chunk goes through an embedding model that turns it into a vector (here, a real MiniLM model producing 384 numbers per chunk). The geometry is the entire trick: texts with similar meaning land near each other, often even when their wording differs; semantic proximity is learned and imperfect. If that sounds like magic, it is the learned kind — an embedding model is trained on enormous amounts of text specifically so that paraphrases end up neighbors. The chunks-with-vectors pile is the index, and it sits there until someone asks something.

Query time, online, every question. Your question goes through the same embedding model in this example. More generally, query and document encoders must share a compatible embedding space; unrelated models cannot be mixed. Then the system computes cosine similarity between the question vector and every chunk vector, sorts, and keeps the top k. Those k chunks get pasted into a prompt under an instruction with the flavor of "Answer ONLY from the context below," followed by your question. The model writes the answer last, after every consequential decision has already been made.

One guard makes the difference between a useful system and a confident one: if even the best score is below a relevance threshold, don't generate. A corpus that contains nothing about your question will still produce a nearest chunk — nearest is a ranking, not an endorsement.

RAG Step by Step

The default run uses a 10-chunk help-center corpus for a fictional smart lamp, k = 3, and the question "My lamp won't connect to my phone." Scores are computed from real embeddings; the answers shown later are authored fixtures, not live model outputs.

1. Index. The help doc splits into 10 chunks — refunds, money transfers between accounts, Bluetooth pairing, firmware, factory reset, two warranty clauses, store credit, the phone app, brightness. Each becomes one vector; on the canvas they land as points in a 2-D projection of that space.
2. Embed the query. The question becomes a vector with the same model and joins the plane as a star.
3. Score. Cosine against all 10 chunks. Now look at what meaning did: the query’s symptom wording differs from "Pair the lamp over Bluetooth: hold the base button five seconds..." — no "connect", no "phone" — and that chunk scores 0.55, rank 1. Meanwhile the chunk that literally contains the word "phone" (the app chunk, about schedules and sunset fades) scores 0.14, rank 5. The pairing cosine is about four times the app cosine here; that ratio is not an accuracy or probability ratio.
4. Select top k. The bracket takes Bluetooth pairing (0.55), factory reset (0.50), and firmware (0.28). Two passengers came along with the winner; that's normal, and harmless here.
5. Augment. The three chunks fly into the prompt, between the system rule and the question.
6. Generate and cite. The model answers — "Hold the base button five seconds until it pulses blue — pairing runs over Bluetooth" — with a citation pointing back at chunk 3. The claim traces to a retrieved line; that's what "grounded" means.

The displayed answer can be supported by the retrieved pairing instruction; the run does not establish what any model knew beforehand.

Why Retrieval Decides the Answer

Switch the preset to Split fact and ask "Can I still get my money back after two months?" The refund-window chunk arrives at rank 1 (0.53) — and it says refunds end at 30 days. The other half of the answer exists in the corpus: unopened items can be exchanged for store credit at any time. But that sentence lives in a chunk that is mostly about gift wrap, and this encoder produces one vector per chunk, so mixing gift wrap with the return condition obscures the latter’s relevance, and the chunk scores 0.24, rank 4. With k = 3, ranks 2 and 3 go to a generic warranty clause (0.29) and a chunk about transferring money between accounts (0.26) — a classic keyword decoy, riding the words "money back". The model, shown only what made the cut, answers: "No — refunds are only issued within 30 days of purchase." Cited and fluent, but incomplete about the available store-credit alternative. A 30-day cash-refund limit and later store credit are different remedies. The exam was failed at the library counter, not at the desk.

Now drag k from 3 to 4 and replay. The store-credit chunk makes the bracket, and the same model with the same question now answers correctly and cites both sources. You repaired a wrong answer without touching the model — which is the lesson's entire thesis in one slider.

The third preset asks "Do you sell replacement bulbs?" Nothing in the corpus is about bulbs, and the scores say so: the best is 0.16, well under the 0.35 threshold, so the system abstains. And you don't have to take the counterfactual on faith — the Inputs panel has a threshold switch. Turn it off and replay: the same all-junk leaderboard now feeds the prompt, the top chunk is a warranty clause, and the model dutifully produces "Replacement bulbs are covered by the two-year warranty" — fluent, cited, and invented. This scripted example illustrates one source of unsupported RAG answers: generating from irrelevant context. It does not measure how often that cause occurs. One toggle separates the honest ending from the confident one.

Two more presets probe the boundary between meaning and words from both sides. Keyword trap asks "Can I trade in an unwanted present through the app?" — the real answer is the store-credit chunk, whose relevant exchange condition uses different wording from the question. But the question contains one baited word, app, and that single word drags the money-transfer chunk to rank 1 by a hair: 0.49 vs 0.48. Embeddings resist keyword pull; they are not immune to it. With k = 1 the model sees only the decoy and cheerfully explains family account transfers; with k ≥ 2 the leaderboard catches the real answer at rank 2 and the reply is grounded — the same one-slider repair as the split fact, for a different disease. Paraphrase asks "How do I erase all my settings and start the lamp like new?" — no chunk says erase, settings, or like new, and the factory-reset chunk still wins at 0.67, the highest top score in the lesson. Meaning carrying the ranking outright, no asterisks.

Edge Cases

• Compatible embedding spaces on both sides. Unrelated encoders produce incompatible coordinates; purpose-trained query/document encoders can be compatible even when they are not identical. Re-embedding the whole corpus is the price of switching models.
• Cosine can go negative. Several chunks here score below zero against unrelated queries — the vectors point away from each other. The leaderboard handles it; intuition should too: negative cosine means opposite directions in this embedding space, not a literal negative amount of shared meaning.
• Near-threshold scores are a judgment call. 0.35 is this corpus's calibrated line, not a law. Real systems tune the threshold per corpus and per model, or replace it with a reranker's verdict. The threshold switch in the Inputs panel is the experiment: run "Not in docs" with it on (honest abstain) and off (confident invention) and compare the two endings — same leaderboard, one guard apart.
• k is a budget, not a virtue. k = 5 on this 10-chunk corpus drags in low-scoring chunks — passengers the model must read past. On the failure preset, k = 4 fixes the answer; k = 10 would fix it too, while teaching the model to swim in noise.
• Chunking is a real design decision. The failure preset exists because one chunk holds two topics. Split too coarse and facts get diluted past retrievability; split too fine and a fact loses the context that made it interpretable.

Common Mistakes

• Debugging the generator when retrieval failed. If the right chunk never made top k, changing generation settings cannot make the omitted chunk available as evidence; the system must retrieve it, otherwise any answer about it lacks that retrieved support. Check the leaderboard first; it decided your answer.
• Reading citations as correctness. A citation points to a source; verify that the source supports the claim and that the retrieved set is sufficient. The split-fact preset produces a wrong answer with a perfectly honest citation.
• Keeping the grep mental model. Retrieval here is nearest-neighbor in meaning-space. The chunk mentioning "phone" lost to the pairing instruction despite the query using different symptom words. If you reason about RAG in terms of keyword overlap, both presets will surprise you in both directions.
• Skipping the abstention branch. A system that must always answer will always find a nearest chunk to misuse. The threshold check is one line of code and most of the difference between "helpful" and "confidently wrong about your own documents".
• Assuming more context is free. Every retrieved chunk spends context-window budget — the same budget the Context & Memory lesson watches fill up. Retrieval is how knowledge gets into the window; it inherits all the window's economics.

A Note on Simplification

The retrieval stack is small enough to inspect. The embeddings are real (MiniLM, 384 dimensions) and every score is true cosine similarity, but the corpus is 10 chunks searched by brute force; production systems hold millions and use approximate-nearest-neighbor indexes (the "vector database"). Production RAG usually adds overlapping chunks, hybrid keyword-plus-vector search, a reranker over the top candidates, and query rewriting. The 2-D plane is only a PCA projection — scores are computed in all 384 dimensions, and the table, not the picture, is the truth. The model's replies are scripted so retrieval, not prose generation, stays under the microscope.