article thumbnail

RAG -- Giving LLMs a Memory

How to Make an AI Answer From Your Data, Not Its Guesses

26 min read
#ai, #rag, #llm, #embeddings, #friday4

Ask a large language model about your company's refund policy, last quarter's sales, or a document you wrote yesterday, and it will do one of two things: admit it doesn't know, or -- worse -- invent a confident, plausible, completely wrong answer. LLMs only know what they saw during training, and they saw none of your data. Retrieval-Augmented Generation (RAG) is the fix, and it has quietly become the default architecture in applied AI. Instead of hoping the model memorized the right fact, you fetch the relevant information at question time and hand it to the model alongside the question. This crash course explains how RAG works, why it beats the alternatives, and how the technique has grown up in 2026.

The Problem RAG Solves

LLMs have three built-in limitations:

You could retrain or fine-tune the model on your data, but that's expensive, slow, and has to be redone every time your data changes. RAG sidesteps all of it: leave the model as-is, and just give it the right context in the prompt. It's the difference between making someone memorize an entire library versus letting them look things up in it.

The Core Idea in One Sentence

Retrieve the most relevant chunks of your data, augment the prompt with them, and let the model generate an answer grounded in that context.

That's the whole trick -- and where the name comes from. The magic is in how you find the relevant chunks, and that's where embeddings come in.

Step 1: Embeddings -- Turning Meaning Into Numbers

The heart of retrieval is the embedding: a model that converts a piece of text into a vector (a long list of numbers -- often 768, 1,536, or more) that captures its meaning. Texts with similar meaning land close together in this high-dimensional space -- even if they share no words.

"How do I reset my password?"     -> [0.021, -0.44, 0.19, ... ]
"I forgot my login credentials"   -> [0.019, -0.41, 0.22, ... ]   (very close!)
"What time does the store open?"   -> [0.55,  0.12, -0.30, ... ]   (far away)

This is why RAG beats plain keyword search: a user asking "my card got declined" can match a document titled "payment failures" even with zero shared keywords, because the meanings are close. Measuring that closeness is usually done with cosine similarity -- the cosine of the angle between two vectors, where a smaller angle means a closer meaning.

This embedding model is a separate, purpose-built model -- not the LLM itself. It's typically small and fast, trained only to produce good vectors, not to generate text.

Step 2: Chunking and Indexing (Done Once, Ahead of Time)

Before anyone asks a question, you prepare your knowledge base:

  1. Collect your documents (PDFs, wiki pages, support tickets, database rows).
  2. Chunk them into bite-sized pieces -- usually a few hundred words each. Too big and retrieval gets noisy; too small and you lose context. Overlapping chunks (a sliding window) preserve meaning across boundaries.
  3. Embed each chunk into a vector using an embedding model.
  4. Store the vectors in a vector database so you can search them fast.
Documents  ->  Chunks  ->  Embedding model  ->  Vectors  ->  Vector store

If you already run PostgreSQL, the pgvector extension can turn it into your vector store (watch for our upcoming deep dive). Dedicated options include Pinecone, Weaviate, Qdrant, Milvus, and Chroma.

A high-leverage trick: before embedding each chunk, prepend a sentence or two describing where it sits in the larger document -- "This is from the Q3 refund-policy section, discussing international orders...". Torn out of its document, a bare chunk is often ambiguous ("it must be returned within 30 days" -- what must?); this contextual retrieval step restores what got lost. Anthropic measured it cutting failed retrievals by about a third on its own, and by roughly two-thirds when combined with keyword search and a reranker (more on both below).

Step 3: Retrieval and Generation (At Question Time)

When a user asks something, the live pipeline runs:

  1. Embed the question with the same embedding model.
  2. Search the vector store for the k nearest chunks -- a similarity search (at scale it's an approximate nearest-neighbor search, trading a sliver of accuracy for a huge speed win). Say, the top 5.
  3. Build a prompt that stuffs those chunks in as context.
  4. Send it to the LLM and return the grounded answer.

The augmented prompt looks something like this:

You are a support assistant. Answer using ONLY the context below.
If the answer isn't in the context, say you don't know.

Context:
---
[Chunk 1: "Refunds are processed within 5-7 business days..."]
[Chunk 2: "To request a refund, go to Orders > ..."]
---

Question: How long do refunds take?

The model now answers from your documented policy -- not from a fuzzy statistical memory. And because you told it to admit ignorance, hallucination drops dramatically.

A Minimal RAG in Pseudocode

# --- Indexing phase (once) ---
chunks = split_into_chunks(load_documents())
for chunk in chunks:
    vector = embed(chunk)              # e.g. an embedding API
    vector_store.add(vector, chunk)

# --- Query phase (per question) ---
def answer(question):
    q_vec = embed(question)
    top_chunks = vector_store.search(q_vec, k=5)   # nearest neighbors
    context = "\n---\n".join(top_chunks)
    prompt = f"""Answer using only this context:
{context}

Question: {question}"""
    return llm.generate(prompt)

That's a working RAG system in ~15 lines of logic. Everything else is refinement.

Naive RAG Is Just the Starting Line

The pipeline above -- embed once, search once, answer once -- is now called naive RAG. It's the right place to start and still handles a large share of real questions. But 2026's production systems have mostly moved past a single retrieval step.

Agentic RAG. Instead of retrieving once before answering, hand the model retrieval as a tool it can call -- the same mechanism behind MCP (which we covered recently) and function calling. Now it can rewrite a vague question into a sharper search query, look at what came back, decide it isn't enough, and search again. A multi-part question ("compare our 2024 and 2025 refund policies") becomes several targeted searches instead of one blurry one. Coding assistants like Claude Code and Cursor work exactly this way: they don't pre-load your whole repo, they go look for the files a task needs, read them, and follow the references.

Adaptive RAG (self-routing). Not every question needs the full machinery. A cheap classifier inspects the query first and picks a lane: a simple fact gets one search; a hard multi-hop question gets the agentic loop; "thanks, that helped" skips retrieval entirely. Cost and latency stay proportional to difficulty.

GraphRAG. Chunk search is great at "find me the passage about X" and bad at "how do all these things connect?" Tools like Microsoft's GraphRAG build a knowledge graph of the entities and relationships across your entire corpus, plus summaries of each cluster, and retrieve over that. It shines on multi-hop and whole-dataset questions ("what themes run through all our support tickets this quarter?"). Indexing a graph costs more -- though newer variants like LightRAG and LazyGraphRAG have cut that sharply -- so it's usually reserved for queries that genuinely need relationship reasoning. (Our pieces on PuppyGraph and Apache AGE cover the graph-database side.)

The throughline: retrieval is becoming a loop the model drives, not a single step that happens before it.

Why Not Just Use a Bigger Context Window?

Modern models accept enormous prompts -- hundreds of thousands, sometimes millions, of tokens. Why not paste in all your documents and skip retrieval? For a small, stable set of documents, that genuinely works, and it's the simplest thing that can. But it breaks down as the corpus grows:

The 2026 view isn't "retrieval vs. a big window" -- it's retrieval feeding the window. Long context lets you be generous: pull the top 20 chunks instead of agonizing over the top 3, and let the model sort it out. A self-route step can decide, per query, whether a quick retrieval is enough or the whole document set should go in. Retrieval is the precision tool; the context window is the workbench it delivers to.

RAG vs. Just Organizing Your Files

Plenty of people already keep a folder of carefully organized Markdown notes and paste the relevant one into an AI chat by hand. That is a form of retrieval -- you're just doing the retrieving. It works beautifully when the collection is small, stable, and you know which file answers which question.

RAG is what you reach for when that stops scaling:

Hand-picked files RAG
Who finds the relevant part You, before you ask The system, from the question's meaning
How much goes in the prompt Whole files Just the matching passages
Ceiling Everything must fit the context window Millions of documents; only the top few chunks are sent
Upkeep Edit the file Re-embed changed chunks; run a vector store
Cost per question You pay for every file you loaded, every time A query embedding plus a handful of chunks

The crossover point is roughly: can I still remember which document to grab, and does it all fit? If yes, organized files -- or a tool that reads your file tree on demand -- are simpler and better. If you find yourself pasting things in "just in case," or you don't know where the answer lives, or the pile changes daily, that's RAG's job.

RAG vs. Fine-Tuning

These solve different problems and are often confused:

RAG Fine-Tuning
Adds new facts ✅ Yes, instantly ⚠️ Awkwardly
Data changes often ✅ Just update the store ❌ Retrain every time
Teaches style/format ⚠️ Limited ✅ Its strength
Cost to update Cheap Expensive
Shows its sources ✅ Yes ❌ No
Setup complexity Moderate High

Rule of thumb: use RAG to change what the model knows, and fine-tuning to change how it behaves. Many production systems use both -- and a third hybrid, RAFT, fine-tunes a model specifically to reason well over retrieved documents.

Where RAG Goes Wrong (and How to Fix It)

RAG is simple to prototype and surprisingly tricky to perfect. The usual failure points:

Measure It, Then Tune It

RAG has many knobs -- chunk size, overlap, k, hybrid weighting, re-ranking -- and tuning them by vibes doesn't scale. Build a small evaluation set of real questions with known-good answers and track three things:

Frameworks like RAGAS automate this scoring (often using an LLM as the judge). Fix retrieval first -- a great generator can't answer from chunks it never received.

RAG in the Real World

You've almost certainly used RAG already:

Using RAG Without Building It

You don't have to write any of the code above to get RAG's benefits today:

A good progression: start with a Project or NotebookLM, move to a framework when you need to customize retrieval, and build the pipeline yourself only when the framework gets in your way.

Quick Reference: The RAG Glossary

Term Meaning
Embedding A vector representing the meaning of text
Chunk A small slice of a document that gets embedded
Vector store A database optimized for similarity search
Similarity search / k-NN Finding the closest vectors to a query (usually approximate at scale)
Cosine similarity The common measure of "how close" two vectors are
Context window How much text you can fit in one prompt
Hybrid search Semantic + keyword (BM25) search combined
Re-ranking A second pass that reorders retrieved results by relevance
Contextual retrieval Prepending a short description of each chunk's context before embedding it
Grounding Tying the answer to retrieved evidence
Agentic RAG The model calls retrieval as a tool, iteratively -- rewriting queries and searching again
Adaptive / self-route RAG Classifying each query to decide how much retrieval it needs (or none)
GraphRAG Retrieval over a knowledge graph of entities and relationships, for multi-hop questions
Faithfulness Whether the answer's claims are actually supported by the retrieved evidence

Getting Started Checklist

✅ Pick an embedding model (an API, or a local one via Ollama) ✅ Choose a vector store -- pgvector if you already run Postgres ✅ Chunk your docs (~300-500 words, with overlap) and embed them ✅ Wire up: embed question → search top-k → stuff prompt → generate ✅ Instruct the model to answer only from context and cite sources ✅ Build a small eval set of real Q&A; track context recall and faithfulness before you tune anything ✅ Start naive (one search, one answer); add hybrid search, then re-ranking, then an agentic loop -- only where the numbers say you need it

Conclusion: RAG turns a general-purpose LLM into an expert on your world -- without retraining anything. The recipe is timeless: convert meaning into vectors, retrieve what's relevant, and let the model reason over it. It's cheaper than fine-tuning, updates the moment your data does, and -- critically -- it can show its work, which is what makes AI trustworthy enough to deploy. The frontier in 2026 is making retrieval something the model does for itself, mid-thought, as many times as a question demands -- but that still rests on the same foundation you can build today. Start small with a folder of documents and a top-5 search, get an answer grounded in your own files, and you'll see why RAG became the default architecture for putting AI to work on real problems.

Enjoyed this article? Share it with someone who'd love it too.

Most covered topics

Databases Python AI PHP Linux JavaScript