Lesson 5.6 · 11 min read
Throughout this module we have been splitting things apart. Ten application servers instead of one. Caches holding copies. Workers running elsewhere. Every time we made a copy of something, we quietly created a possibility we haven't yet confronted: two copies of the truth that disagree.
The cache holds the old profile picture while the database holds the new one. That was a small taste. Now we go to the heart of it, because the moment your data itself lives on more than one machine, disagreement stops being a bug you can patch and becomes a fundamental property of the universe you have to make peace with.
This is the hardest idea in this module, and it is the one that most changes how you think. It's also, unusually for something with a name like "theorem," a deeply practical thing that shows up in product decisions you will make. Take it slowly.
Your data lives on two machines, in two different places, because you want it to survive one of them dying and you want users near each to be served quickly. The two machines constantly copy changes to each other, so they hold the same data. This is replication, and it's how essentially every serious database survives failure.
Now the network between those two machines breaks. Not the machines, which are both perfectly healthy and both receiving requests from users. Just the connection between them. Neither can talk to the other, and, crucially, neither can tell whether the other is dead or merely unreachable. This is called a partition, and it is not exotic. Networks fail constantly: a cable, a router, a bad configuration, a data center losing connectivity for ninety seconds.
Now a user writes to machine A. Something has to happen, and there are exactly two options. Sit with each one.
Machine A accepts the write. It cannot tell machine B, because they can't talk. So machine B now holds old data. A user reading from B gets an answer that is wrong. The system stayed available, and it gave someone incorrect information.
Machine A refuses the write. It says, in effect, "I cannot safely accept this, because I cannot confirm the rest of the system will know about it." The user gets an error. Nothing incorrect was recorded anywhere, and the data everywhere remains consistent, but the system was unavailable to a user who was talking to a perfectly healthy machine.
Read those again and notice that there is no third option. You either accept the write and risk disagreement, or refuse it and deny service. The network is broken and that is not a choice you get to make, so you must choose which of the other two things to sacrifice.
That is the CAP theorem, and it says: in the presence of a network partition, a distributed system must choose between consistency and availability. You cannot have both. Not because engineers haven't tried, but because the situation logically forbids it.
The letters are consistency, availability, and partition tolerance, and the way they're usually described, "pick two of three," is misleading in a way that obscures the actual insight.
Consistency, here, means every reader sees the most recent write, no matter which machine they ask. There is one truth and everyone gets it.
Availability means every request receives a response. The system answers.
Partition tolerance means the system continues to function when the network between its parts fails.
Now, the correction that makes this useful. Partition tolerance is not optional. Networks fail. That is simply true, and it will happen to you, and you cannot decide otherwise. So for any system whose data lives on more than one machine, P is not a choice, it's a fact of the environment.
Which means CAP is not "pick two of three." It is: when a partition happens, and it will, do you choose consistency or availability? That is the whole theorem, and it's a single binary choice about behavior during failure. Everything else about it is people misremembering.
Abstraction dissolves the moment you look at real products, because you've experienced both choices as a user, without knowing what you were experiencing.
Choosing consistency means that during a partition, the system refuses to serve rather than serve something possibly wrong. You have used this. Your bank's app, occasionally, says a transfer cannot be completed right now, please try again. That is a system that would rather be briefly unavailable than briefly wrong, and it is the correct choice, because the alternative is a world where you withdraw the same money twice because two machines each thought you had it.
The class of things where consistency wins is easy to identify: anything where two people acting on stale data causes a real-world contradiction. Money moving. The last seat on a flight. A unique username. An inventory count of one. In each case, allowing two machines to independently say yes produces something that cannot be undone by an apology.
Choosing availability means that during a partition, the system answers with whatever it has, even if that might be slightly out of date, and reconciles later when the network heals. You have used this vastly more often. A social feed that shows a like count of 41 while your friend sees 42. A comment that takes a moment to appear for someone else. A collaborative document where two people's edits merge a second later.
The class of things where availability wins: anything where being briefly out of date is invisible or harmless, and being unavailable is intolerable. Nobody's life is affected by a follower count lagging by two seconds. But an inaccessible social network is a dead one.
This is why the honest form of the question is never "which is better." It is: for this specific piece of data, what does it cost to be wrong for a second, and what does it cost to be unavailable for a second? Answer that, and the architecture follows. And notice that this question is not a technical one. It is a question about the meaning of the data to the person using it, which puts it squarely in your territory.
Systems that choose availability are usually described as eventually consistent, and the phrase is more precise and less alarming than it sounds.
It promises that if writes stop, all copies will converge on the same value, eventually. It does not promise when. It does not promise that a reader won't see old data in the meantime. It does not promise that two readers, at the same instant, see the same thing.
That sounds bad stated coldly, and it's the operating basis for a large portion of the internet, because most data genuinely tolerates it. The window is usually milliseconds. The disagreement is usually invisible. And the payoff is a system that stays up when a data center loses connectivity.
But eventual consistency creates a genuinely new problem: what happens when both machines accepted conflicting writes during the partition, and now they can talk again? Two truths, and only one can survive.
The strategies are all imperfect, and knowing they exist is what protects you from assuming the database has quietly solved this. Last write wins keeps whichever change has the later timestamp, which is simple and silently discards someone's work. Merging applies both changes where the data structure allows it, which works beautifully for something like a set of items and not at all for a single number. Or the system surfaces the conflict and asks a human to resolve it, which you've seen in file sync tools that leave you two versions of a document and a shrug.
The point is not to memorize these. It is to know that a conflict is not resolved by magic, and someone chose the resolution rule, and the rule can lose data. When you build on an eventually consistent system, that rule is part of your product's behavior, whether or not anyone told you.
Here is what most explanations of CAP leave out, and it is the part that will affect you far more often than partitions do.
Partitions are rare. But the tradeoff is not confined to partitions, because consistency has a cost that is paid continuously, in latency, even when everything is perfectly healthy.
Think about what strong consistency requires. Before machine A can confirm a write, it must be certain the other machines know about it. That means talking to them and waiting for their agreement. And you learned in post 0.3 what talking to another machine costs: real time, bounded by physical distance, which is bounded by the speed of light. A write that must be confirmed across three data centers on two continents cannot be fast. It is not a matter of better engineering. It is geography.
So a strongly consistent system pays for its correctness on every single write, forever, in milliseconds. An available system doesn't wait, which is why it's faster, and the price of that speed is the possibility of disagreement.
This is the more useful way to hold the whole idea. You are always trading agreement against speed. During a partition, that trade becomes stark and binary and gets a famous name. The rest of the time, it's a dial, and every distributed database gives you some version of that dial, and turning it toward correctness makes things slower and turning it toward speed makes things looser.
You will probably not choose a distributed database from scratch. But this tradeoff will find you, in three places, and you'll recognize it now.
Every cache is a small CAP decision. The cache is a copy. Serving from it is choosing availability and speed over consistency. The expiration time you chose is you setting the dial, deciding precisely how wrong your data may be. You made a CAP tradeoff in the last post without the vocabulary; now you have it.
Every "eventually" in a product spec is one too. When a feature says the search index updates within a minute, or the analytics dashboard reflects data from an hour ago, or a follower count is approximate, someone chose availability. The right response is not alarm. It is the right question: is a minute acceptable for this, and what happens to a user who acts on stale information during that minute?
And the dangerous case: reads after writes. A user changes their name and immediately reloads. Their write went to one machine; their read went to another, which hasn't caught up. They see the old name and conclude the save failed. They try again. This is one of the most common real bugs in systems that use read replicas for scaling, and it looks exactly like a broken save button. If you ever debug something where "the change didn't stick, but it's there when I check later," you are looking at eventual consistency in the wild, and you now know precisely what you're looking at.
The reason this post sits near the end of the architecture module is that it is where all the tradeoffs stop being separate.
Every good engineering decision in this module has had the same shape. Monolith or microservices: simplicity against independence. Vertical or horizontal scaling: a single powerful thing against many disposable ones. Sessions or tokens: control against scale. Caching: speed against truth. And now, consistency against availability, which is the deepest version of the same pattern, and one that has been formally proven to admit no clever escape.
What separates a design engineer from someone assembling parts is not knowing which option is correct. It's knowing that the choice exists, understanding precisely what each side costs, and being able to say which cost this particular product, with these particular users and this particular data, should pay. That is a judgment, and it requires understanding both the system and the human on the other side of it, which is exactly the pair of things you have been building the ability to hold at once.
There is no perfect database, and there never will be, and now you understand that this is a mathematical fact rather than a temporary inconvenience awaiting a better engineer. What there is, is a correct choice for a specific piece of data, and the person who should be making it is the one who knows what that data means.
Do this before the next post
Take three pieces of data from something you've built, or something you use, and place each one on the dial.
Start with something obvious. A user's follower count, or a view count, or a "last active" timestamp. Ask: if two people looking at this at the same moment saw different numbers, would anything bad happen? Almost certainly not. This data wants availability. It should be cached aggressively, replicated freely, and nobody should ever wait for it.
Now find something where you flinch. A payment. An inventory count where only one item remains. A username being claimed. Ask the same question, and notice how differently it feels. If two machines each independently said yes, what would exist in the world afterward? Two charges. Two people holding the same seat. Two accounts with the same name. This data wants consistency, and it should be willing to be slow, and even to refuse service, rather than be wrong.
Finally, find something in the middle, where it's genuinely a judgment call, and try to argue both sides honestly. A comment count on a post. The number of items in a shopping cart. Someone's online status.
That third exercise is the important one. The first two are easy and the world is mostly made of the third. Being able to sit with a piece of data and reason about how much wrongness it can tolerate, and for how long, and what a user would do if they encountered it, is the practical form of everything in this post. It is not database administration. It is design, conducted at a layer most designers never reach.