Lesson 5.2 · 11 min read
Here is a piece of code. Read it and decide whether it's correct.
Someone books the last seat on a flight. The code checks how many seats remain. There is one. So it creates the booking and decrements the count to zero.
It is obviously correct. You can read it, follow it, and satisfy yourself completely. It is also broken, and it will sell that seat twice, and it will do so only when two people book at the same instant, which is precisely the moment when the seat is most likely to be booked twice.
This post is about that class of bug. It is the hardest kind to find, because the code is correct when you read it, correct when you test it, and correct every single time you use it yourself. It fails only under concurrency, which is to say, only in production, only under load, only when it matters.
And it is the single area where AI-generated code is most reliably and most confidently wrong, because a model producing that booking function is producing exactly what the request described.
The last post established that load means requests in flight simultaneously. Now look at what "simultaneously" does to code.
Your booking function does three things: read the seat count, decide, write. Between the read and the write there is a gap. It might be two milliseconds. During that gap, the code holds a belief about the world, that there is one seat left, and that belief is not protected.
Two requests arrive a millisecond apart. The first reads: one seat. The second reads: one seat, because the first hasn't written yet. The first decides to book. The second decides to book. The first writes zero. The second writes zero. Two bookings, one seat, and a count that says zero, which is consistent with everything except reality.
This is a race condition: the outcome depends on the relative timing of things that happen at once. And notice what the shape is, because you will meet this shape everywhere once you can see it. Read, decide, write, with a gap. It's called check-then-act, and it is broken whenever anything else can act during the gap.
You have already met this pattern without recognizing it. Check whether this username is taken, then create the account. Check whether the user has enough balance, then charge them. Check whether this file exists, then create it. Check whether the invitation was already used, then use it. Every one of these is the same bug wearing different clothes, and every one of them appears in production code written by careful people, because there is no moment while writing it where anything feels wrong.
Ordinarily you'd say: write a test. But consider what a test does. It calls the function once, checks the result, and passes.
To catch this bug a test would have to call the function twice, from two different places, arranging for the second to arrive precisely inside the gap between the first one's read and write. You can construct such a test with effort, and almost nobody does.
So the bug ships. And then it doesn't happen for six weeks, because the gap is two milliseconds wide and traffic is light. Then you get popular, or a sale starts, or someone writes a script, and suddenly two requests land in the same two milliseconds and the seat is sold twice. Somebody reports it. You cannot reproduce it. The logs show two perfectly ordinary successful requests.
This is why concurrency has to be reasoned about rather than discovered. It is not a bug you find by being careful in the ordinary ways. It's a bug you avoid by recognizing the shape of the code that causes it, before you write it, and reaching for one of the mechanisms below.
Post 3.4 called transactions the crown jewel of relational databases and moved on. Time to say what one actually is.
A transaction is a group of database operations treated as a single, indivisible unit. Either every operation in it takes effect, or none of them do. There is no partial state, no matter what fails, no matter when.
The canonical example is moving money: subtract from one account, add to another. If the second operation fails after the first succeeded, money has vanished. A transaction makes that impossible. Both, or neither.
Two things it gives you, and they're different, and people conflate them.
Atomicity is that all-or-nothing property. Your multi-step operation cannot be interrupted halfway and leave the world in a state that shouldn't exist. If your booking creates a row in bookings and updates a row in flights, and both are in one transaction, then a crash between them leaves neither. This is enormously valuable and it is not, by itself, enough to fix our bug.
Isolation governs what concurrent transactions can see of each other. This is the property that actually addresses races, and it's the one people don't know exists. Databases offer several isolation levels, which are, precisely, a dial trading strictness against speed. At weaker levels, more concurrency is permitted and more anomalies are possible. At the strictest, transactions behave as though they ran one after another, in some order, with no overlap at all, which is exactly what you want and costs the most.
Here's the part that matters for you. The default isolation level in most databases is not the strictest one. So wrapping your booking in a transaction, which most people believe fixes it, often does not. Both transactions read one seat, both write, both commit, and atomicity was preserved perfectly while the seat was still sold twice. Atomicity protects you against partial failure. It does not protect you against a stale read.
That distinction is worth more than any other sentence in this post, because it is the misunderstanding that produces the bug in code written by people who thought they had handled it.
To close the gap, something must prevent the second request from acting on what it read while the first is still deciding. There are two philosophies, and they are the same two you find everywhere in this field.
Pessimistic locking assumes conflict and prevents it. When the first transaction reads the seat count, it tells the database to lock that row. The second transaction, arriving and trying to read the same row, is made to wait. It waits until the first commits, then reads the updated value, sees zero seats, and correctly refuses. No double booking, ever.
This is expressed in SQL as reading a row "for update," which means: give me this row, and don't let anyone else touch it until I'm done. It is simple, correct, and it costs you: the second request waited. Under heavy contention on the same row, everyone queues, and throughput on that row collapses to one at a time.
Optimistic locking assumes conflict is rare and detects it. Each row carries a version number. You read the row and its version. You do your thinking. When you write, you say: update this row, but only if the version is still what I read. If someone else changed it in the meantime, the version has moved, zero rows match, and your update fails. You then retry, reading the fresh value.
Nobody waits. Conflicts are caught rather than prevented. This is much better when conflicts are rare, and worse when they're common, because a losing writer does its work twice.
You know this dial. It's pessimistic-versus-optimistic, prevention-versus-detection, and it's the same reasoning as consistency-versus-availability, which is the next post. Contention high, correctness critical, lock. Contention low, throughput matters, version and retry.
And for the narrow, common case there is something better than either. Make the operation atomic in the database itself. Rather than reading the count, deciding in your code, and writing it back, issue one statement: decrement the seat count by one, where the count is greater than zero. The database performs the check and the change as a single indivisible operation. If it affected zero rows, there were no seats and you refuse the booking. There is no gap, because there is no moment between the read and the write. There is only one operation.
That is the deepest lesson here, and it generalizes. The gap exists because you pulled the data out of the database, thought about it, and put it back. Whenever the decision can be expressed as a single statement that the database evaluates and applies at once, the race disappears rather than being managed. Prefer that whenever it's possible, and it's possible more often than people realize.
One special case, because it's the most common check-then-act in existence and it has a clean answer.
Check whether this email is registered. It isn't. Create the user. Two simultaneous signups with the same email both check, both find nothing, both create. Now you have two accounts with one email, and every piece of code you ever write that assumes email is unique is quietly wrong.
The answer is not to check more carefully. It's to make the database enforce it, with a unique constraint on the column. Now the second insert fails, at the database, atomically, regardless of timing, because the database is the one place where the check and the write are the same act.
Your code changes shape as a result, and this is worth noticing. Instead of checking and then acting, you act and handle the failure. Try to insert; if the database rejects it for violating uniqueness, tell the user the email is taken. This feels backwards to most people the first time, and it is the only version that is actually correct.
The principle generalizes to every constraint the database can hold: uniqueness, foreign keys, checks that a value is in range, that a balance is never negative. A rule enforced by the database cannot be violated by a race, because the database is the single serialization point where concurrent things become ordered. A rule enforced in your application code can always be violated by two copies of that code running at once, and remember from post 5.1 that there are ten copies of your application, on ten machines, none of which know about each other.
That last sentence is the whole reason this post lives in the architecture module. The moment you scaled horizontally, every check your application performs became a check performed by ten independent things that cannot see each other. The only thing they share is the database. So the database is where truth must be enforced.
You are not going to reason about isolation levels daily. Here is what actually matters, and it's a small number of instincts.
Recognize check-then-act on sight. Whenever you read something, make a decision based on it, and then write, ask what happens if someone else does the same thing between your read and your write. If the answer is bad, you have work to do. This recognition is the entire skill.
Prefer a single atomic statement. Push the check into the write. UPDATE ... WHERE count > 0 rather than reading, comparing, and updating.
Push invariants into the database. Anything that must always be true, uniqueness, non-negative balances, valid references, should be a constraint, not a code path. Code paths run ten times in parallel. Constraints do not.
Use transactions for atomicity, and know that they are not automatically enough for isolation. If two transactions can read the same row and both act on it, name the isolation level or lock the row explicitly.
And when reviewing what an AI writes, look for the gap. A model asked to write a booking function will write the readable, obvious, broken version, because the readable obvious version is what exists in a million tutorials, and because the request, "book a seat if one is available," describes check-then-act exactly. The code will be correct. The code will pass its tests. The code will sell the seat twice on the day you finally get traffic, which is the worst possible day.
You are the one who knows that two people might click at once. That has never been a fact about code. It's a fact about people, which is your territory, and it happens to be the thing the machine writing your code cannot see.
Do this before the next post
Find a check-then-act in your own work. You have one. Look for anywhere you read a value, decide something, and write. Claiming a username. Applying a coupon. Decrementing anything. Marking an invitation used. Incrementing a counter.
For each one, walk it slowly, out loud, with two users. User A reads. User B reads. User A writes. User B writes. Say what the world looks like afterwards. Half the time it's fine, because nothing depended on the value being current. The other half, you have just found a real bug that has been sitting in your code the entire time, waiting for two people.
Then fix one, and fix it the good way. If it's a uniqueness check, delete the check and add a constraint, and handle the rejection. If it's a decrement, replace the read-decide-write with a single conditional update, and check whether it affected a row. Notice, when you're done, that the code is shorter and that you have moved the guarantee from a place where it was hoped to a place where it is enforced.
That move, from hoping to enforcing, is what this post is for, and it is the difference between software that works when you use it and software that works when the world does.