Lesson 5.4 · 11 min read
Here's a moment you've experienced from both sides without thinking about it.
You upload a video. The site says "we're processing it, we'll email you when it's ready," and lets you carry on. Or you place an order, and the confirmation appears instantly, but the receipt email arrives forty seconds later. Or you click "export my data" and get told it'll take a few minutes.
Compare that to the alternative, which is what a beginner's system actually does. You upload a video and the page hangs, spinning, for four minutes while the server transcodes it, and if your connection blips, everything is lost and you start over.
The difference between those two experiences is the subject of this post, and it's one of the most important architectural ideas in the module. Some work simply cannot happen inside a request, and understanding why, and what to do instead, is the difference between a system that survives contact with real users and one that falls over the first time somebody uploads something large.
Recall from the scalability post what a request actually costs. While a request is in flight, it occupies a connection, holds memory, and consumes a slice of your server's finite capacity to handle simultaneous work. And in-flight requests multiply: the longer each one takes, the more of them exist at once, and the closer you sit to the cliff.
Now put a four-minute video transcode inside a request. That request holds resources for four minutes. Ten uploads and you have ten long-running requests going nowhere, while ordinary users trying to load a page queue behind them. Your entire application becomes unresponsive because a handful of people uploaded videos, which is a deeply stupid way to fail and a completely typical one.
It's worse than that, because a request is also fragile in ways short requests never reveal. The user's browser will give up waiting. So will the load balancer, which has a timeout, and will simply sever the connection. If the server restarts during a deploy, the work vanishes mid-flight with nothing recorded. And if it fails at minute three, there is no mechanism to try again, because the thing that was trying is gone.
So the honest rule: a request should do the smallest amount of work necessary to answer the user, and nothing else. Everything else has to happen somewhere that isn't a request.
Synchronous means the caller waits for the work to finish before moving on. You ask, you wait, you get the answer. Everything we've built so far in this series is synchronous: a request arrives, the server queries the database, the answer comes back, the response is sent.
Asynchronous means the caller doesn't wait. The work is handed off to happen later, and the caller carries on immediately.
This distinction is not about programming syntax. It's an architectural choice about who waits, and it maps directly onto a question about your product: does the user need the result of this work in order to continue?
They need to know their order was placed. They do not need to have received the confirmation email. They need to know their video was accepted. They do not need it to be transcoded. They need to know their account was created. They do not need the welcome email sent, the analytics event recorded, or their profile picture resized into six formats.
Draw that line correctly and your system becomes fast and durable. Draw it wrong and users wait for things they never needed.
So you've decided some work happens later. Where does "later" live?
A queue is a durable list of work waiting to be done. When your server decides something needs doing, it doesn't do it. It writes a message describing the job, "transcode video 8492," "send order confirmation to this address," and drops it into the queue. That takes about a millisecond. Then it responds to the user immediately, and the request is finished, and the resources are released.
Separately, elsewhere, a different set of processes called workers watch that queue. A worker picks up a message, does the actual work, and when it succeeds, removes the message from the queue. Workers are not serving user requests. They have no browser waiting on them, no timeout, no impatient human. They can take four minutes. They can take an hour.
The queue in the middle is what makes this work, and it has properties worth stating precisely.
It's durable. The message survives. If every worker dies, the queue still holds the jobs, and when workers come back, the work gets done. Nothing is lost because the server restarted.
It absorbs bursts. Ten thousand people upload at once? Ten thousand messages land in the queue instantly. The workers churn through them steadily. Your system doesn't fall over; it forms an orderly line and gets slower to finish, which is a wonderful failure mode compared to collapsing. This is the coffee shop from the scalability post, except the queue is explicit and the customers went home rather than standing in the doorway.
It decouples. The part of your system that decides work must happen is now completely separate from the part that does it. They don't know about each other. They can be deployed separately, scaled separately, and fail separately. If transcoding is slow, you add more transcoding workers, and nothing else in your system changes at all.
That last property is the real prize. Scaling becomes surgical. Rather than scaling your whole application because one job type is heavy, you scale the workers for that job type.
Because workers process jobs independent of any user, a set of problems appears that synchronous code never has to think about, and they're all genuinely interesting.
A job fails. The email service is down. The video is corrupt. What now? The great advantage here is that you can simply try again, because the message is still in the queue. Retries are the fundamental gift of asynchronous work: a failure is not a failure, it's a "later." Almost always you retry with increasing delays, a second, then five, then thirty, then five minutes, so that a struggling downstream service isn't hammered while it's trying to recover. This is called backoff, and it is the difference between helping a service recover and finishing it off.
A job fails forever. Some jobs are simply never going to succeed. The video really is corrupt. Retrying it a million times is worse than useless, because it consumes workers that should be doing real work. So after some number of attempts, the message is moved out of the main queue into a dead letter queue, a holding area for jobs that failed permanently. This is where you look when someone says "my export never arrived," and it's one of the more useful things to know exists.
A job runs twice. This one is the subtle one and it will happen to you. A worker picks up a message, does the work successfully, and then crashes before it can mark the message as done. The queue, quite reasonably, assumes the job was never completed and hands it to another worker. The work happens twice.
Most queues guarantee at-least-once delivery, meaning a message will be processed, possibly more than once. Guaranteeing exactly-once is famously difficult and generally not offered. So the burden lands on you, and it has a name: your jobs should be idempotent, which means running them twice has the same effect as running them once.
That word sounds academic and the idea is entirely practical. Sending an email is not idempotent: run it twice, the user gets two emails. Charging a credit card is very much not idempotent, and this is how people get charged twice, and it is one of the most common serious bugs in payment systems. Setting a user's status to "active" is idempotent: do it a hundred times, nothing changes after the first.
The usual fix is to make the job check before acting. Before charging, check whether a charge for this order id already exists. Before sending, check whether this email was already recorded as sent. You're not preventing the double execution; you're making it harmless. When you see production code with a check that looks redundant, "if already processed, return," this is what it's for, and it is not redundant at all.
Queues protect you from slow work. But your request path still calls things, a database, a cache, another service, and each of those can be slow or absent. Two ideas handle it, and both are absent from most code written quickly.
A timeout is not optional. Every call to something external must have a limit on how long you will wait. Without one, a hung dependency means your request waits forever, holding resources, and the pile-up from post 5.1 begins. A request that fails after two seconds is enormously better than one that waits for thirty, because the failure is bounded and the resource is released. When you read code that calls another service and see no timeout specified, you have found a way for someone else's bad day to become an outage.
And decide, deliberately, what happens on failure. Some dependencies are essential: if the database is gone, there is no answer, and the honest response is an error. Many are not. If the cache is down, you go to the database and everything is slow but correct. If the recommendations service is down, you show the page without recommendations. If the analytics call fails, nobody should ever know.
This is graceful degradation: the system loses a limb rather than dying. It is entirely a design decision, and it's yours. For each thing a page depends on, ask what the user should see if that thing is unavailable. Usually the answer is "slightly less, and no error." Almost always the code, as written, shows them a blank screen instead, because nobody asked the question.
You returned instantly. The work is happening somewhere. How does the user find out?
The simplest answer, and often correct: they don't, immediately. The email arrives when it arrives. The analytics event was recorded and nobody needs to know. Most asynchronous work is genuinely invisible.
When they do need to know, there are a few honest approaches, and you've seen all of them. The interface polls, asking every couple of seconds whether the job is done, which is crude and works fine. Or the server pushes an update over a persistent connection when the work completes, which is what makes progress bars and live status updates feel alive. Or you simply notify them out of band, by email, later, which is the right answer for anything genuinely long.
Notice this is a design problem and not merely a technical one. The moment work moves out of the request, you have created a state your interface must represent: pending. The user needs to understand that their thing was accepted, is happening, and isn't finished. Doing that well, honestly, without either lying about progress or leaving people uncertain, is exactly the kind of problem that sits between design and engineering, and belongs to whoever can see both sides.
So what goes in a queue? The rule, applied honestly:
Anything slow. Video processing, image resizing, report generation, large exports, complex calculations. If it takes more than a second or two, it does not belong in a request.
Anything that talks to an external service. Sending email, sending SMS, calling a third-party API, hitting a payment provider's webhook. You do not control whether that service is fast, or up. Putting it in the request path means their outage becomes your outage. Putting it in a queue means their outage becomes a slightly delayed email.
Anything the user doesn't need in order to continue. Analytics, notifications, cache warming, search index updates, syncing to another system.
And especially, anything that calls an AI model. This is worth stating explicitly, because it is the most relevant instance of this pattern in modern building and the one most often gotten wrong. A model call can take many seconds. It can fail. It's rate limited. It's expensive. It is a slow, unreliable, external, costly operation, which means it ticks every single box above. AI features that generate something substantial belong in queues, with retries, with a pending state in the interface, and with idempotency so a retry doesn't bill you twice. Building an AI feature that calls a model synchronously inside a request is one of the most common architectural mistakes being made right now, and you now know exactly why it will hurt.
What stays synchronous? Reading data the user is looking at. Writing the thing they just submitted. Authentication. Anything where the answer is the point of the request.
Do this before the next post
Take something you've built and find everything happening inside a request that shouldn't be.
Go through your endpoints and, for each one, list what happens before the response is sent. Then interrogate each item with the one question that matters: does the user need this to have finished, in order to continue?
You will find things. An email being sent. An external API being called. Something being resized, or logged to a third party, or recomputed. Each one is currently holding your user's request open, tying up a slice of your server's capacity, and coupling your uptime to somebody else's.
For each one, ask the follow-up questions this post gives you. If this ran twice, what would happen? If it failed, would anyone ever know, or would it fail silently forever? If the external service it depends on went down for an hour, what would your users experience?
You don't have to build a queue today. What you must be able to do is look at a request and see clearly which parts of it are the answer to the user's question, and which parts are work that merely happens to be sitting there, holding everything up, waiting for a reason to fail.