Lesson 6.1 · 12 min read
Everything in this series so far describes a kind of software that has existed for decades. Requests and responses, databases and caches, queues and containers. The concepts are mature. The tradeoffs are well understood, and mostly settled.
This module is different, because the thing it describes is genuinely new. Not new as in fashionable. New as in: the properties of this component are unlike anything else in the systems you have learned to build, and the architectural patterns for handling it are being worked out right now, in public, by people who are frequently getting it wrong.
That's an opportunity for you, and a real one. The gap between "I use AI tools" and "I can engineer a product that has AI inside it" is enormous, and comparatively few people have crossed it, because it requires exactly the combination this series has been building: understanding systems well enough to see where this strange new component fits, and understanding users well enough to know what it should be doing for them.
We begin by taking the chatbot out of your head.
Strip away everything you've experienced as a user, and consider what an LLM actually is when it appears in a system you're building.
It is a service you send a request to, over HTTP, with an API key in the header, containing some text. After some time, it sends back a response containing some text. That's it. It is an external API, exactly like the payments API and the maps API and the email API from post 2.1. You already know how to call it. You already know where the key must live, which is on your server, never in the frontend.
Nothing about that is novel, and it's worth saying plainly, because a great deal of confusion comes from people treating this as a magical new category rather than as a box on the diagram with arrows going in and out.
What is novel is the character of that box. Every external service you've integrated so far has been fast, cheap, reliable, and deterministic. It returned the same answer for the same question, it took forty milliseconds, it cost a fraction of a cent, and if you sent it valid input, you got valid output.
An LLM violates every one of those properties. And each violation forces an architectural response. That's the whole post: five ways this component is strange, and what each one demands of you.
A model call does not take forty milliseconds. It takes seconds. For a long response, tens of seconds. The reason is structural rather than incidental: the model generates its answer one piece at a time, each piece depending on the ones before it, so a long answer is fundamentally a long sequence of steps that cannot be parallelized away.
Now apply what you know. From post 5.1: slowness multiplies into load. A request that holds a connection for eight seconds is holding your server's resources for eight seconds. From post 5.3: anything slow does not belong inside a request.
So the architectural conclusion is immediate and non-negotiable, and I will state it as bluntly as I can: if a model call takes meaningful time and the user isn't watching it happen, it belongs in a queue. Generating a summary, processing an upload, enriching a record: these are jobs, with workers, with retries, with a pending state in the interface. Putting them inline in a request is the same mistake as transcoding video inline, and it fails the same way, at the same moment, for the same reason.
But there's a case where the user is watching, and it deserves its own treatment, because it explains something you've seen a thousand times.
When you talk to a chat interface, the words appear one at a time. That is not a design flourish. It is the architecture leaking through, honestly. The model produces text incrementally, and rather than making you wait ten seconds for the whole thing, the system streams it to you as it arrives, over a connection that stays open. The total time is identical. The perceived time is transformed, because you begin reading at four hundred milliseconds instead of at ten seconds.
This is one of the purest examples in modern software of an architectural choice and a design choice being the same choice. Streaming exists because the model is slow, and it works because a human reading is also slow, and the two slownesses can be made to overlap. As someone who thinks about experience, you should recognize what happened: the latency didn't go away. It was hidden inside an activity the user was going to spend time on anyway.
Every other API you've called is effectively free at small scale. An LLM is not. Every call costs real money, and the cost is proportional to the amount of text going in and coming out.
Text is measured in tokens, which are chunks of text roughly corresponding to word fragments. A page of text is perhaps five hundred tokens. You pay for tokens in and tokens out, and the price varies enormously by model.
The architectural consequences are things you have already learned, arriving in a new context.
Rate limiting is not optional. Recall from the API security post: an endpoint with no rate limit and a per-request cost can be hammered until your bill explodes. That was a general warning. Here it is a specific, urgent one, because this is the case where it actually happens. An unprotected endpoint that calls a model is a bill generator that anyone on the internet can operate.
Caching becomes financially interesting. From post 5.2: cache things that are expensive to produce and frequently requested. An LLM response is the most expensive thing in your system to produce. If two users ask the same question, and the answer is not personal, serving the second from cache saves you money as well as seconds. The economics of caching, which used to be about servers, are now about invoices.
The size of what you send is a cost, not just a latency. This is genuinely new. In every other system, sending more data cost you bandwidth and time. Here it costs you money, directly, per call, forever. Which brings us to the constraint that shapes AI architecture more than any other.
This is the one that most changes how you design, and the one most people misunderstand.
A model remembers nothing between calls. Nothing. Each request arrives at a service that has no idea you have ever spoken before. It is, precisely, the stateless server from post 5.1, taken to its logical extreme.
So how do chat interfaces remember your conversation? They don't, and the model certainly doesn't. On every single message you send, the application sends the entire conversation so far back to the model, along with your new message. The memory is not in the model. The memory is in your application, which replays the history on every call.
Sit with that, because it explains almost everything downstream. Every "AI that remembers" is an application storing things in a database and choosing what to re-send. The remembering is engineering, not intelligence.
And there is a ceiling. The context window is the maximum amount of text a model can consider in one call, everything you send plus everything it generates. It is large in modern models, and it is finite, and you will hit it.
Which means the central engineering problem of building with LLMs is this: you have a limited, expensive space, and you must decide what goes in it. Not everything fits. The conversation history grows without bound; the window does not. The user's documents are enormous; the window is not.
So what do you do? You select. You put in the window the things most relevant to answering this particular question, and you leave out everything else. And the moment you say that sentence out loud, you have described a search problem: given a question, find the most relevant pieces of a large corpus. That is why vector databases and retrieval exist, and why they get their own post later in this module. They are not an AI feature. They are a response to a hard limit on a shared resource, which is the most ordinary kind of engineering problem there is.
The conversation history has the same shape. Past some length you cannot send it all, so you must summarize the older parts, or drop them, or store them and retrieve the relevant bits. Every product with a long-running AI conversation has made this choice, and every one of them has a strategy, and users experience that strategy as the AI's personality and reliability.
Send the same input to the same model twice and you may get different output. This is by design.
Every architectural instinct you have built is quietly premised on determinism. Tests assert that a function returns an expected value. Caches assume the same key maps to the same answer. Retries assume that trying again produces the same result. Debugging assumes you can reproduce the problem.
None of those hold cleanly here, and each one needs adjusting rather than abandoning.
Testing can no longer assert exact output. It has to assert properties: is the response valid JSON, does it contain the required fields, is it in the right language, does it avoid saying the things it must never say. This is the beginning of evaluation, which is a whole discipline and which we'll return to in this module. But note the shift: you're testing a distribution of behaviors, not a value.
Reproducing a bug requires you to have captured exactly what was sent, because you cannot simply run it again and see. If you don't record the input, the output, and the version of the model, a report of "it said something wrong yesterday" is unfalsifiable and uninvestigable. This is the single strongest argument for the observability post later in this module, and it's why AI systems need instrumentation that ordinary systems can survive without.
And retries mean something different. Retrying a failed database write repeats an identical operation. Retrying a model call may produce something better or worse. That's occasionally useful, and it means a retry is not free of consequence, so idempotency, from the queues post, has to be thought about again: if the job that calls the model runs twice, does the user get two summaries, and are you billed twice?
Every other component in your architecture fails loudly. A database returns an error. An API returns a 500. A build fails. The failure is visible, catchable, and unambiguous.
A model fails by producing something plausible and incorrect, formatted exactly like a correct answer, with no error, no exception, no status code, and no signal whatsoever that anything went wrong. Your code cannot catch it because from the code's perspective nothing happened. A 200 came back. There was text in it.
This is a category of failure your entire training in software has not prepared you for, and it is why building AI products is genuinely harder than it looks.
The architectural response is to stop trusting the output and start checking it, mechanically, wherever you can.
If you asked for structured data, validate its structure before using it. Do not assume it returned valid JSON with the required fields. Check, and have a path for when it didn't. If you asked for a value from a fixed set, verify it's actually one of them. If the model's output is going to be used to take an action, deleting something, sending something, spending something, then a human should be in that loop, because a plausible-but-wrong action is a real-world event that no apology reverses.
And the deepest response is a design one, which is yours to own. Design the interface so that being wrong is survivable. Show the source alongside the claim. Make the output a draft the user edits rather than a fact the system commits. Make the destructive action require confirmation. The system's fallibility is not something to hide behind clever prompting. It is a property of the component, and interfaces built on top of it must be honest about it. That is not a limitation you're working around. It's the actual design problem, and it belongs to whoever understands both the component and the person.
So where does the box go on the diagram?
An LLM is an external service, so it lives outside your system, reached by an arrow that leaves the boundary. Which means, from the diagram post: their outage is your outage, unless something protects you.
That something is a queue, so the arrow to the model should usually come from a worker, not from an API server. Retries with backoff for when it's overloaded, a dead letter queue for what fails permanently, idempotency so a retry doesn't double-bill. All of this is post 5.3, applied without modification, because a model call is simply the most extreme case of a slow, unreliable, expensive external dependency.
Around it sits a cache, because responses cost money. In front of it sits a rate limiter, because the endpoint is a bill generator. Behind it, feeding it, is your own storage, holding the conversation history and the documents, along with the logic that decides which fragments of them are worth the space in the context window on this particular call.
Every piece of that is something you learned before this module started. That is the point of the module's placement. The AI is not a new kind of architecture. It is a strange new component that makes the architecture you already understand mandatory, because it is slow enough, expensive enough, and unreliable enough that you cannot get away with the shortcuts that ordinary components forgive.
Do this before the next post
Take an AI feature, one you've built or one you use, and locate it in an architecture you draw yourself.
Draw the client, your server, and the model as a box outside your boundary. Now answer, with your finger on the diagram: does the request path go straight from a user request to the model? If it does, count what that means. How long is that request open? What happens when the model is slow? What happens when it's down? What stops one user from calling that endpoint two thousand times?
Then find the context. What text, exactly, is being sent to the model on each call? Where does it come from? If it includes a conversation history, what happens on the hundredth message? If it includes a document, what happens when the document is four hundred pages?
Finally, sit with the fifth strangeness, and ask the question that most people never ask. When this model returns something confidently wrong, and it will, what happens next? Does your code notice? Does the user notice? Is the wrong answer displayed as a fact, or offered as a draft? Is it about to cause an action that cannot be taken back?
If you can answer that last question well, you are already doing something most teams shipping AI products right now are not, and the reason you can is that you understand what the component actually is: not a mind, not a feature, but a slow, expensive, forgetful, non-deterministic, confidently fallible service, sitting at the end of an arrow on your diagram, waiting to be engineered around.