Lesson 6.6 · 9 min read
Back in post 6.1 we ended on a problem and promised to return to it. The model's context window is finite and expensive. Your knowledge is not. A company's documentation is millions of words; the window holds a fraction of that. So on any given question, something must decide which few pages are worth the space.
That is a search problem, and it is the least glamorous and most important part of most AI products. This post is about why ordinary search is inadequate for it, what replaced it, and what the replacement actually costs.
The obvious approach is to search your documents for the words in the question, which is what search has done for decades and does well.
It breaks in a specific and fatal way. A user asks, how do I cancel my subscription? The relevant document is titled "Terminating your plan" and never uses the words cancel or subscription. Keyword search finds nothing. Or it does the opposite: a document mentioning cancel forty times in an unrelated legal context ranks first.
The problem is that keyword search matches strings, and the user asked about meaning. Cancel and terminate are different strings and nearly the same idea. Every synonym, every paraphrase, every way of asking the same thing differently, is invisible to a system comparing characters.
What you actually need is a way to search by meaning: give me the passages that are about what this question is about, whatever words either happens to use. For most of computing's history there was no good way to do that, because meaning is not a thing computers hold.
Now there is, and understanding it is genuinely worth your time, because it's one of the few ideas in this module that is conceptually new rather than architecturally new.
An embedding is a list of numbers that represents a piece of text, produced by a model trained so that texts with similar meanings get similar numbers.
Think of it as coordinates. A point on a map has two numbers, and points close together are places near each other. An embedding has hundreds or thousands of numbers, so it's a point in a space with hundreds of dimensions, which you cannot picture and do not need to. The property that matters survives: texts that mean similar things land near each other.
"How do I cancel my subscription" and "terminating your plan" share almost no words, and their embeddings sit close together, because the model that produced them was trained on enough language to place them by meaning rather than by spelling. "How do I cancel my subscription" and "the recipe calls for two eggs" land far apart.
So meaning becomes geometry, and finding relevant text becomes finding nearby points. That's the trick, entire. It sounds like it shouldn't work and it works remarkably well.
To use it, you do two things. Before anything else, you take all your documents, cut them into passages, embed each one, and store the resulting points. Then, when a question arrives, you embed the question the same way, and look for the stored points nearest to it. Those are your relevant passages, found by meaning, in milliseconds.
You now have a few million points and you need the ones nearest to a given point. This is where a normal database fails you, and the reason is worth understanding rather than accepting.
A relational database is superb at finding rows where a column equals a value, because it can build an index, a structure that lets it skip almost everything. That works because "equals" divides the data cleanly.
"Nearest to" does not. There is no way to index a thousand-dimensional space such that you can leap directly to the neighbors of an arbitrary point. Done exactly, finding the nearest points means comparing the query against every single stored point, which is fine for ten thousand and hopeless for fifty million.
So vector databases do something interesting: they give up on exactness. They use algorithms for approximate nearest neighbor search, which find almost certainly the closest points, almost always, very fast, by cleverly organizing the space so that most of it can be ignored. You accept that occasionally the true best match is missed. In exchange you get an answer in milliseconds instead of minutes.
That tradeoff, correctness for speed, should feel familiar by now. It's the same shape as caching and as the availability side of CAP. Notice too that a vector database is, exactly as post 3.4 described, a specialist holding a derived copy. The documents themselves still live in your real database or object storage. The vectors are an index built from them, and if you lost the vector database entirely, nothing of value would be gone, only recomputable.
Assemble the pieces and you have the architecture that most AI products on documents actually use, called retrieval augmented generation, or RAG. The name is intimidating and the thing is simple.
A question arrives. Embed it. Search the vector database for the most similar passages. Take the top few. Put them into the prompt, along with the question and an instruction along the lines of: answer using the information provided, and say so if it isn't there. Send it to the model. Return the answer.
That's RAG. Retrieve, then generate. It exists because the model has no knowledge of your private documents and no room to be shown all of them, so you find the small relevant part and show it that.
Two things worth being clear about, because both are commonly misunderstood.
This is not teaching the model anything. The model remains exactly as stateless as post 6.1 said. You are not training it, updating it, or giving it memory. You are putting some text in a prompt. Next call, that text is gone unless you put it there again. All "memory" and "knowledge" in these systems is retrieval, performed by your code, on every single call.
And retrieval quality is the ceiling on answer quality. If the right passage is not among the ones you retrieved, the model cannot use it. It will answer from what it was given, or from general knowledge, fluently and wrongly. When a RAG system gives a bad answer, the fault is usually not the model. It's that the search returned the wrong passages, and nobody looked.
This is why the observability post matters here specifically. The retrieved passages must be recorded in the trace, or you cannot distinguish "the model reasoned badly about good information" from "the model reasoned perfectly about the wrong documents." Those need entirely different fixes, and they look identical from outside.
RAG demonstrates beautifully and disappoints in production, reliably, and the reasons are worth knowing before you meet them.
Chunking. Documents must be cut into passages before embedding, and where you cut them shapes everything. Too small, and a passage lacks the context to be understood: a paragraph explaining an exception to a rule stated three paragraphs earlier is worse than useless alone. Too large, and its embedding averages several topics into a muddy point near nothing in particular. There is no correct answer, only tradeoffs, and this unglamorous decision does more to determine system quality than the choice of model.
Structure is destroyed. A table becomes a run-on sentence. A document's hierarchy vanishes. The fact that this paragraph appeared under a heading saying deprecated is gone, and the model confidently explains the deprecated approach.
Similar is not relevant. Embeddings find passages about the topic. They do not find the passage that answers the question. Ask about the current refund policy and you may retrieve five passages discussing refund policies, including two obsolete ones. The system has no idea which is current, because recency and authority are not properties of meaning. This is why serious systems combine vector search with keyword search, filter by metadata like date and source, and often re-rank the retrieved set with a second, slower model that judges actual relevance rather than mere similarity.
And nothing tells you when retrieval fails. No error, no low-confidence signal, no gap. The model receives irrelevant passages and produces a fluent answer anyway. Silent failure again, exactly as in 6.1, arriving through a different door.
Step back and look at what has been built here, because the shape is clarifying.
Embeddings turn meaning into geometry. Vector databases make geometry searchable at scale by trading exactness for speed. RAG uses that search to solve a resource allocation problem: a small, costly context window, and far more knowledge than fits.
None of that is intelligence. It's search, storage, and a budget. The retrieval half of a RAG system is closer in spirit to a database index than to anything cognitive, and treating it that way, as infrastructure to be measured, tuned, and instrumented, is what separates a system that works from a demo that impressed people once.
Which returns us to where this module began. The AI is a strange component: slow, expensive, forgetful, and confidently fallible. Everything around it, the queues, the caches, the rate limits, the versioned prompts, the traces, the evaluation sets, and now the retrieval layer, exists to make something dependable out of it.
That work is engineering, and most of it is the ordinary engineering you learned in the five modules before this one. The reason it is worth learning properly is that the extraordinary part, the model, is available to everyone. The difference between products is entirely in the unremarkable machinery built around it, and in whether someone was paying attention to what the person on the other end actually needed.
Two exercises, one small, one that will teach you more than the whole post.
First, get an intuition for embeddings. Ask any AI to explain, with a short concrete example, why "the cat sat on the mat" and "a feline rested on the rug" would have similar embeddings while "the cat sat on the mat" and "the cat is not on the mat" would too, despite meaning opposite things. That second case is not a trick question. It is one of the real limitations of similarity search, and thinking about why it happens will teach you more about embeddings than any diagram.
Second, take a set of documents you know well and design the retrieval, on paper, before writing anything. Where would you cut them into passages, and what breaks at each cut? What happens to your tables? How would you know that a retrieved passage is out of date? If a user asked a question whose answer spans two distant sections, would you retrieve both?
Then write down five real questions users would ask, and for each, name the passage that ought to be returned. That is an evaluation set for retrieval, and it is the thing that will tell you whether your system works, separately from whether the model sounds convincing.
Almost nobody does this, and it is the difference between a RAG system that answers questions and one that produces plausible text about approximately the right subject. You now know why, and you know which one you're building.