Lesson 5.3 · 10 min read
At the end of the last post, we left you with a problem. You scaled your application servers horizontally, made them stateless and disposable, put a load balancer in front, and then all ten of them turned around and pointed at one database. The bottleneck moved rather than disappearing.
Now we start solving it, and we start with the most powerful tool available, one that is almost embarrassingly simple in principle. If the database is the constraint, the obvious move is to stop asking it things. Most of what your database is doing, right now, in whatever you've built, is answering the same question over and over and producing the same answer every time. Ten thousand people load your homepage; ten thousand identical queries run; ten thousand identical results come back.
Caching is the practice of remembering an answer so you don't have to compute it again. That's the whole idea. And it is, without much competition, the highest-leverage performance technique in software, capable of taking a query that runs in eighty milliseconds and making it take one, which as you now know from the last post is not a fortieth of the latency but roughly eighty times the capacity.
It is also the technique most likely to hand you a subtle, maddening bug, for reasons that are structural rather than accidental. Both halves of that sentence are this post.
A cache is a fast, temporary store that holds copies of results, keyed by the question that produced them.
The pattern is always the same, and it's worth burning into memory because you'll see it everywhere:
Something asks for data. You check the cache first. If the answer is there, you return it immediately, and you never touch the slow thing at all. That's a cache hit. If it's not there, you do the expensive work, you store the result in the cache for next time, and you return it. That's a cache miss.
The first person to load the page pays the full cost. Everyone after them gets the answer nearly instantly, from memory, without the database being involved. Your hit rate, the fraction of requests served from cache, is the number that matters. A ninety percent hit rate means your database is doing one tenth of the work it was doing, which is not a marginal improvement, it is the difference between one database and ten.
Where does the cache live? Almost always in memory rather than on disk, because memory is orders of magnitude faster, and because a cache is allowed to be lost. That's a real design freedom. The truth is in the database. The cache is a convenience, and if it vanishes, nothing is destroyed. Everything just gets slower for a moment while it refills. This is precisely why the key-value stores from the storage post exist, and why "fast but you can afford to lose it" was the phrase used to describe them. Now you see what they were for.
The word "cache" gets used for one specific thing, and it's actually a strategy applied at every layer of the journey you learned in post 0.1. Seeing all of them at once is genuinely clarifying, because it shows that caching isn't a feature you add, it's a principle that appears wherever there is a slow thing and a repeated question.
The browser caches. When you visit a site and it downloads images, fonts, and scripts, your browser keeps them. Reload the page and those files never travel again. This is why the second load is faster, which you observed with your own eyes in the network tab back in Module 0.
The CDN caches. From the latency post: copies of your static files sitting on servers near your users, all over the world. A CDN is a geographic cache, and its entire purpose is that a user in Jakarta gets an answer from Jakarta rather than from Virginia.
The application caches. Your server keeps computed results in a shared memory store, so ten servers can all benefit from work any one of them did.
The database caches. Databases hold recently-used data in memory internally, so a repeated query often never reaches the disk.
Four layers, one idea. And notice the shape: each layer is trying to stop a request from reaching the next, slower layer. Caching is a series of walls, each one catching some fraction of the traffic before it descends. The best request is the one served by the browser, which never left the machine. The second best is the one served by the CDN, which never reached your servers. And so on down.
Here is where it gets genuinely difficult, and where every experienced engineer has a scar.
A cache holds a copy of the truth. The truth then changes. The cache does not know this. Now your cache holds a lie, and it will confidently serve that lie to everyone who asks, quickly and cheerfully, until something tells it to stop.
A user updates their profile picture. The new one is saved to the database. But the old one is in the cache, and the cache is what's being served, so they refresh the page and see their old picture and conclude your product is broken. Nothing is broken. The truth changed in one place and the copy didn't.
This is called cache invalidation, and it means removing or updating a cached entry when the underlying data changes. It has a reputation for difficulty that is entirely earned. The reason is worth stating plainly: the code that changes the data is usually nowhere near the code that cached it. Someone updates a user's name in an account settings endpoint. Somewhere else, entirely unrelated, a homepage feed cached a rendered card containing that name. Nobody remembers the connection exists. The name changes; the card doesn't. And this happens quietly, without error, in production, for one user, and someone reports it three weeks later.
There are a few honest strategies, and each is a real trade rather than a solution.
Expiration by time. Every cached item is given a lifetime, and after it passes, the item is discarded and the next request recomputes it. Set a feed to expire after sixty seconds, and stale data can exist for at most a minute. This is simple, robust, requires no coordination, and is by far the most common approach. Its cost is precisely stated in the trade: you have accepted that your data can be up to sixty seconds wrong. For a feed, that's fine. For an account balance, it is not. The correct expiration time is a product decision about how stale a given piece of information may acceptably be, and it should be made by someone who understands both the data and the user, which is you.
Explicit invalidation on write. When the underlying data changes, you actively delete the cached copy. Correct and immediate. And fragile, for the reason above: it requires that everyone who writes data knows every cache that might hold a copy of it, forever, including caches added next year by someone who has never read your code.
Writing through the cache. When you update the data, you update the cached copy at the same moment, so they never diverge. Cleaner where it applies, though it only helps for the specific entries you know to update.
Which is why the field's most repeated joke has real teeth. There are two hard problems in computer science: naming things, cache invalidation, and off-by-one errors. The joke is a joke. The middle item is not.
The judgment, which is the actual skill.
Cache things that are expensive to produce and frequently requested. Both conditions. Something expensive that's asked for once a day isn't worth caching; something trivial asked constantly isn't worth caching either. The value is the product of the two.
Cache things that many users share. A trending list, a homepage, a product catalog, a leaderboard: one cached copy serves everyone, so the hit rate is enormous. Caching something unique to a single user gives you a hit rate of, at best, that user's own repeat visits, which is a much thinner return.
Cache things where staleness is tolerable. This is the real filter, and it's a product question, not a technical one. Can a follower count be thirty seconds old? Almost certainly. Can a bank balance? Absolutely not. Can a "seats remaining" number on a booking page? Now you're in an interesting conversation about whether you'd rather be fast or be right, and that conversation is exactly what this module exists to prepare you for.
Do not cache things that are personal and sensitive without extreme care about how they're keyed. A cache keyed carelessly, on the URL rather than on the URL plus the user's identity, will happily serve one user's private data to another user who requested the same URL. This is not a hypothetical, it is one of the more spectacular ways real companies have leaked data, and it happens because a cache is a shared thing and someone forgot to include identity in the key. If a cached response varies by user, the user must be part of the key. Always.
And the practical reality behind hit rates: caches are finite. When full, they evict something to make room, usually whatever was used least recently. So your cache naturally fills with what's popular, which is why hit rates in real systems are often surprisingly good. Popularity is concentrated. A small number of things are asked for constantly, and those are exactly the things a cache will keep.
Caching is where you stop tuning a feature and start designing a system, and it's a genuine turning point in how you think.
You now understand why real applications are fast, which is not because their code is clever but because most requests never reach the expensive part. You understand why a CDN was in the latency post, why key-value stores were in the storage post, and why they were always the same idea. And you understand the real cost, which is not memory or money but the risk of serving something that isn't true, and that this cost is paid in correctness, and that deciding how much of it to pay is a product decision.
When you build with AI and it adds caching to make something fast, you now have the question to ask, and it is not "does this work." It is: when this underlying data changes, what makes this cached copy go away? If the answer is "nothing," you have just introduced a bug that will appear in production, for some users, unpredictably, weeks from now, and it will be extremely difficult to trace. If the answer is "it expires in five minutes," then the real question is whether five minutes of wrongness is acceptable here, and only you can answer that, because it depends entirely on what the data means to the person reading it.
That question, asked every time, is the whole discipline.
Do this before the next post
Go find the caches you're already using without knowing it.
Open any site, open the network tab, and reload it twice. On the second load, look at the size column. Many requests will say something like "from cache" or "from memory" instead of a size. Those files never traveled. You are looking at the browser cache doing its job, and it has been doing it for you your entire life.
Then find something in what you've built that is expensive and repeated. A query that runs on every page load and returns the same thing for everyone. A count that's computed by scanning a whole table. An external API call you make on every request for data that changes hourly.
For that one thing, answer three questions on paper. How stale can this be before a user is meaningfully misled? What would remove it from the cache when the underlying data changes? And if you cached it keyed only on the URL, could one user ever receive another user's copy?
If you can answer all three, you're ready to cache it, and you'll do it correctly. If you can't answer the second or the third, you've just discovered why caching is the technique that separates people who make things fast from people who make things fast and wrong.