Lesson 6.5 · 10 min read
A user tells you the AI feature gave them a wrong answer yesterday afternoon.
In an ordinary system, this is a tractable problem. You find the request in the logs, see the error, read the stack trace, reproduce it locally, and fix it. The system is deterministic, so running the same input again produces the same failure, which you can then watch happen.
Here, none of that works. There was no error. The request succeeded, returned a 200, and produced fluent text that happened to be wrong. Running it again might produce something perfectly correct. And you cannot even run it again, because you do not have the input: the prompt was assembled at runtime from a template, some conversation history, three retrieved documents, and the user's message, and you saved none of it.
You have a bug report you cannot reproduce, about a failure that raised no error, in a system whose behavior you cannot reconstruct. This is not an edge case. This is the normal condition of an AI system that was not instrumented, and it is why observability is not optional here in the way it arguably is elsewhere.
Traditional monitoring answers questions about health. Is the service up, how many requests per second, what's the error rate, how long do requests take. These are still true and still useful, and an AI system needs all of them.
They just don't answer any of the questions you actually have.
Your error rate is zero. Every call succeeded. That is precisely the problem: the failure mode of a model is a successful response containing something wrong. A dashboard showing all green is compatible with a system that has been confidently lying to users for a week.
So the questions that matter are different in kind. What exactly was sent to the model? Which prompt version, which model version? Why did it choose that tool? Which documents were retrieved, and were they the right ones? Where did the twelve seconds go? What did this cost? And, hardest: is the quality of the output getting better or worse over time?
None of those are answered by uptime and latency. They require recording what happened inside the request, in detail, as a matter of routine.
The core idea is a trace, and it comes from ordinary distributed systems, where a single request touches many services and you need to see the whole path.
A trace is the complete record of one operation, from start to finish. Within it, each distinct step is a span: a named unit of work with a start time, an end time, and some attached information. Spans nest, because work nests. A trace is therefore a tree, and reading it shows you both the sequence and the structure of what occurred.
For an AI system, one trace covers one user interaction, and the spans are the things that happened inside it:
Retrieve relevant documents, taking two hundred milliseconds, returning five results. Assemble the prompt, using template version 7. Call the model, taking six seconds, consuming four thousand input tokens and eight hundred output, costing some amount. Parse the response and validate it against the expected schema. Call a tool because the model asked. Call the model again with the tool's result. Return the answer.
Each of those is a span, with the inputs and outputs recorded on it. And now the trace is the explanation. When someone reports a bad answer, you open its trace and see, in order, what was retrieved, what was sent, what came back, and where the time and money went. The question "why did it say that" becomes answerable, because you kept the evidence.
For an agent, this matters even more, because a run is a loop and each turn is a span, and the trace is a readable transcript of the machine's reasoning: it looked here, decided this, called that, got an error, tried something else. Reading an agent trace is the closest thing to watching it think, and it is the only way to understand why a fifteen-step run went wrong at step four.
Precision here, because the difference between useful and useless instrumentation is a handful of fields.
The exact input to the model. Not the template. The final, assembled text, with everything filled in. This is the single most important thing, and the one most often missing, because it feels redundant to store something you could theoretically reconstruct. You cannot reconstruct it. The history was different then, the retrieval returned different documents, the template has since changed.
The exact output. Before any parsing or cleanup.
Every version. Which prompt version, which model version, which retrieval configuration. This is what turns "outputs got worse on Tuesday" from an anecdote into a diff, and it is why the versioning discipline from the prompts post exists.
Tokens and cost. Per call, so it can be aggregated per feature, per user, per day. AI is the first component in most systems where an individual request has a meaningful, variable price, and where a change to a prompt can double your bill without changing a single line of code.
Latency, per span. Because "it's slow" is useless and "retrieval takes two seconds and the model takes six" tells you where to look. Usually the answer is surprising.
Tool calls, in agent systems. Which tool, what arguments, what came back, especially errors. The errors are where agents go wrong.
And a way to link a trace to a user's report. A trace identifier, surfaced somewhere, so that "the answer at 3pm was wrong" becomes a specific trace rather than a search through a haystack.
There is a real tension here, and it should be named. Recording model inputs and outputs means recording user data, sometimes sensitive user data, in a place engineers can read it. This is a genuine privacy obligation. It requires decisions about redaction, retention, and access, and those decisions must be made deliberately, before you turn this on, and not discovered afterward.
Observability tells you what happened. Evaluation tells you whether it was any good, and it is the discipline that separates teams who improve their AI systems from teams who fiddle with prompts and hope.
The problem it solves is one from post 6.1. You cannot write a test asserting the output equals an expected string, because the output legitimately varies. So instead you assert properties, and you assert them across many cases rather than one.
Some properties are mechanical and cheap. Is it valid JSON. Does it contain the required fields. Is the category one of the permitted values. Is it under the length limit. Did it avoid mentioning a competitor. These catch a startling proportion of real failures, and they cost nothing to run.
Some require a reference answer. You keep a set of inputs with known-good outputs, and measure whether the system gets them right. This is expensive to build and enormously valuable, and the best source is production: every failure a user reports becomes a case in the set, forever, so you can never regress on a bug you have already fixed.
And some require judgment, at which point teams often use a model to judge another model's output against criteria. This works better than it has any right to, and it has an obvious weakness worth holding onto: you are evaluating a fallible system with a fallible system, and if both share a blind spot, the evaluation will confirm the failure. It is a useful tool, not an oracle.
Here is the point of all this, and it's the sentence to remember. Without evaluation, you cannot tell whether a prompt change made things better. You will change the wording, check three examples that look nicer, ship it, and silently degrade a case you did not think to check. Every team that has built an AI product has done this. The ones who stop doing it are the ones who built a set of cases and ran against it before shipping.
The workflow that results is unglamorous and effective. A user reports a bad output. You open its trace. You see the exact input. You add it to the evaluation set with what the answer should have been. You change the prompt. You run the whole set. You see that you fixed this and broke nothing else, or that you fixed this and broke two other things, which is genuinely common. Then you ship, with the version recorded, so you can see whether reality agrees.
That loop is what building an AI product actually consists of, and every step of it depends on having recorded what happened.
There is a category of platform for this, doing roughly what this post describes: capturing traces of model calls, letting you inspect the exact inputs and outputs, tracking cost and latency, storing evaluation sets and running them, and comparing versions.
I'm not going to organize this post around any of them, for the same reason as the prompts post. This part of the ecosystem is young, the names will change, and a post that taught a product would be a post with an expiry date. The concepts have no expiry date.
Judge whatever you use by whether you can answer these. For a specific bad output a user reported, can I see the exact text sent to the model, and the exact response, and which prompt and model version produced it? Can I see where the time went, and what it cost? For an agent, can I read every step it took? Do I have a set of cases I run before shipping a change? And can I tell, honestly, whether last month's version was better than this month's?
If yes, your tooling is adequate. If no, the gap is not a product you're missing. It's a decision nobody made.
Two things make instrumentation more essential here than anywhere else in this series, and both follow from the nature of the component.
The system changes underneath you. Providers update models. Your prompts were tuned against behavior that is now slightly different. Your code is unchanged, your tests pass, and your product behaves differently. Without a record of versions and a set of cases to run, you will not notice, and you will not be able to explain it when someone else does.
And failures are silent. Every other outage in this series announces itself. A model failure looks like success, produces no error, degrades no metric, and reaches a user who quietly loses trust and does not file a report. The only way such a failure ever becomes visible is if someone deliberately built the apparatus to see it.
That apparatus is not overhead on top of the real work. In a system whose central component is non-deterministic and confidently fallible, the apparatus is how you know anything at all, and building it is not a sign of maturity so much as the entry fee for operating this kind of software honestly.
Do this before the next post
Take an AI feature, one you built or one you're planning, and answer the question this post is named for. If a user tells you it gave a wrong answer yesterday, what can you actually find out?
Walk it concretely. Can you locate that specific interaction? Can you see the exact text that was sent to the model, including whatever was retrieved and whatever history was included? Can you see what came back? Can you say which version of the prompt was in use, and which model, and whether either has changed since?
For most people reading this, the honest answer will be no to nearly all of it, and that's expected. The instructive part is noticing what it would take to make each one a yes, and how small each of those changes is compared to how much they'd give you.
Then do the one thing that is worth doing today, whatever else you do. Start a file, anywhere, and write down five real inputs to your AI feature along with what the correct output would be. That file is an evaluation set. It is the beginning of being able to tell whether you are making things better or worse, and until it exists, you genuinely cannot, no matter how carefully you read the outputs. Five cases is enough to start, and starting is the entire difficulty.