Lesson 4.5 · 13 min read
Something is wrong. The site is down, or a feature that worked yesterday doesn't, or a deploy went red, or a user has sent you a screenshot of an error you've never seen. Your heart rate has gone up, and you're about to do the single worst thing available, which is to start changing code and hoping.
This post is about the alternative. There is a method for this, and it isn't complicated, and having it means the difference between a calm twenty minutes and a frantic afternoon. Every experienced engineer has this method, mostly without ever having been taught it explicitly, absorbed by watching others. You're going to be handed it directly.
The method rests on one idea. Before you fix anything, find out where the problem lives. Almost all wasted debugging time is spent fixing things in the wrong place, and almost all of that comes from skipping the step where you actually find out.
Before any of the method, one thing takes priority, and it is the instinct that most separates a professional from a beginner.
If users are looking at something broken right now, your first action is not to diagnose. It is to make it not broken. Roll back to the last working version. That's what rollback is for, and it takes seconds.
The reason is not just kindness to your users, though it is that. It's that debugging under pressure, with people affected and someone asking for updates, is when you make your worst decisions. You skip steps. You change three things at once. You deploy an untested fix that makes it worse. Rolling back converts an emergency into a problem, and problems can be solved carefully.
So: users affected? Roll back. Now you have a working site, no audience, and all the time you need. Then begin.
When something is broken, it is broken in exactly one of four places, and identifying which one takes about two minutes. This is the whole method, and everything else in this post is detail.
The build. Your code never became a deployable thing. Nothing new was released. The old version is still running.
The configuration. The code built and deployed, but it's running with wrong or missing values, so it can't connect to something, or it's connecting to the wrong thing.
The server. The backend is running but a request to it is failing. The code is there, executing, and something in it is going wrong.
The client. The backend is fine and answered correctly, but the browser can't display it, because the frontend code errored.
These four are sequential, which is enormously convenient, because it means you can walk them in order and stop when you find the problem. A failed build means nothing after it even ran. Bad configuration means the server may never have started. If the server didn't answer, the client's behavior is a consequence, not a cause.
The single most valuable question, before anything else: did the deploy succeed? If it didn't, you are in the first place. If it did, you are in one of the other three. That's your first fork, and it costs one glance.
Open the deploy log on your platform. It's long and mostly noise, and people are intimidated by it because they try to read it like prose. Don't. A build log is read backwards and by shape.
Scroll to the bottom. The last thing that happened is nearly always the thing that broke. Everything above it is history that went fine.
Find the first actual error, not the last line. This is the part people get wrong. When a build fails, it often produces a cascade: one real failure, followed by twenty consequential complaints. The last line is usually something useless like "command failed with exit code 1," which tells you nothing. Scroll up from the bottom until you find the first line that describes an actual problem. That's your bug. Everything below it is the sound of the building falling down after the load-bearing wall came out.
Note which stage it died in, because the stages have completely different causes. Installing dependencies failed? Something is wrong with the packages: a package that doesn't exist, a version conflict, a private package the build server can't authenticate to. Building failed? Something is wrong with your code: a syntax error, a missing import, a type error, a reference to a variable that doesn't exist. The stages are announced in the log, and knowing which one you're in eliminates most of the possibility space instantly.
The most common build failures are boringly consistent, and you should recognize them by sight. A module that can't be found, which almost always means you imported something you never installed, or you installed it only on your machine and never saved it to the project's dependency list. A syntax or type error that your editor was quietly tolerating but the build is not. An environment variable that was needed at build time and wasn't there. And, memorably, a filename case mismatch: your machine treats Button.jsx and button.jsx as the same file, and the Linux server building your code emphatically does not. That one produces a "module not found" for a file you can see with your own eyes, and it has cost every developer alive at least one bewildered hour.
Remember the reframe from the last post. A failed build means the broken thing was never released. Your users are still on the last good version. Nothing is on fire. You have all the time in the world to read the log properly.
This is the harder and more common case. The deploy went green. The site is up. And something is wrong anyway. Now you're in configuration, server, or client, and you need to find out which.
Start with the client, because it's free to check and takes ten seconds. Open the site, open DevTools, and look at the console and network tab, exactly as Module 1 taught you. You are asking two questions, in this order.
Did the request even go out, and what came back? Open the network tab and find the request that should have happened. This single observation splits the entire problem space:
If there's no request at all, the frontend never asked. The problem is in the client, before the request: a broken event handler, a condition that never fired, a component that crashed before reaching the fetch.
If there's a request that got a 4xx status, the server answered and rejected you. A 401 means it doesn't believe you're authenticated. A 403 means it knows who you are and says no. A 404 means the endpoint doesn't exist at that address, which in production very often means the frontend is calling a URL that's right locally and wrong in production, a configuration problem. These are honest answers from a working server.
If it's a 5xx, the server broke while handling your request. The problem is in the backend, definitively, and no amount of frontend investigation will help. Go read the server logs.
If it's a 200 with the correct data sitting right there in the response, and the screen is still wrong, then everything worked except the frontend's handling of the answer. The bug is in the client, after the response. Check the console for an error.
Notice how much four numbers just told you. This is why the network tab is the most valuable panel in the browser: it locates the problem before you've read a single line of code.
If the server is failing, you need to see what it saw. Every hosting platform gives you the running logs of your backend, and they are a live stream of what your code printed and what errors it hit.
The rules are the same as before, and they apply to any log anywhere. Read from the bottom. Find the first error in a cascade, not the last. Look for the stack trace, which names the file and line where things went wrong, and read the top of it, which is where the error actually occurred.
What you're hunting for, in rough order of likelihood in a fresh production deployment:
A missing environment variable. The code tried to use a value that wasn't supplied, so it got nothing, and then tried to connect to nowhere, or authenticate with nothing. This is the single most common cause of "it works locally and not in production," because your machine had a .env file and the server was never given the same values.
A connection failure. Cannot reach the database. The address is wrong, or the credentials are wrong, or the production database doesn't permit connections from where your server is running. Configuration, again.
An unhandled case. The code assumed something that isn't true in production. A field that's always present in your test data is null for a real user. A list you assumed had items is empty. Real data is messier than your data, always, and this is where that fact announces itself.
A permissions error. The server tried to do something it isn't allowed to do, which in a database with row level security is very often exactly the thing that post warned you about.
Something worth internalizing about production logs: they are only as useful as what your code chose to print. When you're building, adding a log line before and after risky operations, and printing the values you're about to depend on, costs nothing and is the only reason a future emergency will be readable. Log the things you'd want to know at 2am. You are writing a letter to yourself, and you will be grateful.
The method above finds the problem. These habits keep you from making things worse while you do it, and they're the ones that come from experience, usually painfully.
Change one thing at a time. When you're stuck, there's a powerful temptation to change four things simultaneously because one of them might work. If it then works, you don't know why, which means you haven't fixed it, you've hidden it. And if it doesn't, you've now got four new variables. One change, one test, always.
Ask what changed. Things that were working and stopped, stopped because something changed. Almost always. What deployed recently? What configuration was touched? Did a dependency update? Did someone change a database rule? The answer is nearly always in the diff between working and not working, and finding that diff is faster than reading code. This is the single highest-yield question in all of debugging.
Reproduce it before you fix it. If you can't make the bug happen on demand, you cannot know whether you fixed it. You'll change something, the bug won't appear, and you'll declare victory over a problem that's still there, waiting. Get it to happen reliably first. Everything after that is easy.
Distrust your assumptions, and check them. You assume the environment variable is set. You assume the request is being sent. You assume the data has the shape you expect. Every one of those is checkable in about five seconds, and one of them is wrong. That's why the bug exists: reality disagrees with you somewhere, and you can't reason your way to where. Go look.
Read the error message. All of it. Slowly. It has been sitting there the whole time, describing the problem precisely, while you scrolled past it. This sounds condescending and it is nonetheless the most frequently violated rule in software.
Everything above assumes somebody told you something was wrong. That is a poor way to find out, and it means every incident begins with a user having a bad time.
The alternative is that the system tells you. Logs record what happened, one event at a time, and are how you investigate. Metrics are numbers measured continuously over time, and are how you notice. Requests per second. Error rate. How long the ninety-fifth percentile of requests takes. Queue depth. They are cheap to record and they show you shape and trend, which logs cannot.
The percentile matters more than you'd guess. An average response time hides everything: if ninety-five percent of requests take a hundred milliseconds and five percent take twelve seconds, the average looks fine and one in twenty of your users is having a miserable time. Always look at the slow tail, because that's where real people live.
Alerts are metrics with a threshold and someone's attention attached. Error rate above one percent for five minutes. Queue depth growing without bound. The discipline is to alert on symptoms users feel, not on causes, and to alert rarely. An alert that fires often teaches everyone to ignore alerts, and then the real one arrives and is dismissed like all the others. A page that wakes someone at 3am must be worth waking them for.
You do not need this on the project you built last week. You need to know it exists, so that when something matters, the question "how would we find out about this without a user telling us" gets asked before the answer is discovered the hard way.
Paste an error into an AI and it will very often tell you exactly what's wrong. This is one of the genuinely transformative uses of these tools, and you should reach for it without hesitation. AI is superb at reading a stack trace and recognizing a pattern it has seen ten thousand times.
But be precise about the division of labor, because it matters here more than almost anywhere.
AI is excellent at explaining an error and proposing a fix. It is not able to know where the problem lives, because that requires observations only you can make: whether the request went out, what status came back, whether the deploy succeeded, what changed recently, what the logs actually say. Hand an AI a symptom and it will guess, plausibly and often wrongly. Hand it a located problem and it will solve it.
So the shape of a good collaboration is: you locate, AI fixes. You do the two minutes of looking that identifies which of the four places you're in and what the evidence says. Then you bring that: "the deploy succeeded, the request returns a 500, and the server log shows a null reference on this line, here's the function." That gets a correct answer immediately. Whereas "my app is broken" gets a tour of plausible causes, one of which might be right.
And there is one thing AI must never do here, which you should hold firmly. It cannot decide to change something in production. When a system is broken and pressure is high, an AI will confidently propose modifying a database, disabling a security policy, or deploying a fix straight to production. Any of those can turn a broken feature into a permanent catastrophe. Roll back, reproduce somewhere safe, fix, test, then deploy. The order is not bureaucracy. It's the order that prevents you from destroying real data at the worst possible moment, when you're stressed and moving fast.
Users affected? Roll back first. Emergency becomes problem.
Did the deploy succeed? No? Read the build log from the bottom, find the first real error, note which stage. Nothing was released, so relax.
It deployed? Open DevTools. Did the request go out? What status came back? Four numbers tell you whether you're in the client before the request, the server, or the client after the response.
Server broke? Read its logs the same way. Suspect a missing environment variable first, then a connection, then an assumption about real data.
Then, always: what changed? Reproduce it. Change one thing. Read the error properly.
That's it. It fits in a paragraph, it works on every application ever built, and it is the difference between an outage that takes twenty calm minutes and one that eats your evening.
Break something on purpose, somewhere safe, and practice the method on a problem whose answer you already know. That last part is the trick: you learn the process far better when you aren't also anxious about the answer.
In a project that doesn't matter, delete an environment variable from your hosting platform and redeploy. Watch what happens. Does the build succeed? Almost certainly yes, so you're past the first fork. Now open the site. Where does it fail? What does the network tab show? What status comes back? Go read the server log and find the moment it tried to use a value that wasn't there. You have just walked the entire method, top to bottom, on a bug you planted, and it took five minutes.
Then do it again with a different kind of failure. Import a package you never installed and push it, and watch the build fail, and practice reading the log backwards to find the first real error rather than the last useless line. Point your frontend at an API endpoint that doesn't exist and watch a clean 404 appear in the network tab, telling you precisely where the problem is.
Three planted bugs, fifteen minutes, and you will have experienced each of the four places with your own hands. The next time it happens for real, unplanned and at a bad moment, you will not feel the panic. You'll feel something much better, which is recognition.