How RAG (Retrieval-Augmented Generation) actually works

Ask a general-purpose AI model a question about your company's internal handbook, a lecture you recorded, or a spreadsheet you built last week, and it has a problem: it's never seen that content. It was trained on a huge slice of the public internet up to some cutoff date, and nothing you're asking about was in there. So it does one of two things — it says it doesn't know, or worse, it guesses confidently and gets it wrong. That second failure mode has a name: hallucination.

Retrieval-Augmented Generation, or RAG, is the standard fix. The idea is simple to state and a little more involved to build correctly: instead of asking the model to answer from memory, you find the specific passages that are actually relevant to the question, hand those to the model as context, and ask it to answer using only that material. The model isn't guessing anymore — it's summarizing and reasoning over text you gave it a few hundred milliseconds ago.

The four steps

1. Chunking. You can't hand an entire PDF or a two-hour video transcript to a model in one go — there's a limit to how much text fits in a single request, and even within that limit, stuffing in everything makes it harder for the model to find the relevant needle in the haystack. So the source content gets split into smaller pieces first. A PDF gets split by page or section; a video transcript gets split by time window, with each chunk tagged with the timestamp it came from. The chunk size matters more than people expect — too big, and retrieval gets fuzzy; too small, and you lose context that spans a chunk boundary.

2. Embedding. Each chunk gets converted into a vector — a list of numbers, typically a few hundred dimensions — using an embedding model trained specifically so that chunks with similar meaning end up with similar vectors, regardless of the exact words used. This is what makes semantic search possible: a chunk about "revenue growth" and a question about "did sales go up" can match even though they don't share a single word in common.

3. Retrieval. When a question comes in, it gets embedded the same way, and the system searches for the stored chunks whose vectors are closest to the question's vector — this is a nearest-neighbor search over a vector index, commonly done with a library like FAISS. The top handful of matches get pulled out as context.

4. Generation. The retrieved chunks, along with the original question, get assembled into a prompt and sent to a language model, with explicit instructions to answer only from the provided context and to say so if the answer isn't there. The model's job has shifted from "recall a fact" to "read this and summarize it" — a much easier, much more reliable task.

Where it gets harder than the textbook version

Pure vector search has a specific, well-documented blind spot: it's good at meaning, bad at exact details. Ask "what happened in 2017" against a document that also mentions 2016 and 2019, and a pure embedding comparison can genuinely struggle to tell those years apart — to the model, "2017" and "2019" are nearly the same point in vector space, because the surrounding words are so similar. The fix is hybrid retrieval: run a literal, exact-match search for things like years, quoted phrases, or section numbers alongside the vector search, and merge the results. It sounds like a small detail, but it's the difference between a demo that works on easy questions and a system that holds up under real ones.

The other thing that trips people up is treating every question the same way. "Summarize this document" and "what does section 3.2 say" are fundamentally different asks — the first needs the whole document's context, the second needs one specific passage. A system that always retrieves the same number of chunks for every question will do a mediocre job at both. Detecting broad, whole-document questions and routing them differently — sending full text instead of a handful of retrieved chunks — closes that gap.

Why this matters if you're building one for a project

If you're building a RAG project for a college submission, this is exactly the kind of design decision an examiner will probe: not "does it work," but "why did you chunk it that way," "why FAISS and not something else," "what happens when the question is ambiguous." Understanding the four steps well enough to explain why each one exists — not just that it exists — is what separates a project you can defend from one you're hoping nobody asks too many questions about.