Not Black Friday. Just a Tuesday
The worst outage I’ve seen didn’t happen on any of the “official” peak days. Ordinary Tuesday, mid-month, an appliance store. Around 7:20pm, right as evening ad traffic started climbing, the payment gateway began timing out. The site was technically up — pages loaded, the cart calculated fine. Checkout just silently failed. Nobody noticed for another forty minutes, because monitoring only checked whether the homepage responded, not whether payments actually went through.
Black Friday is the easy case — you know the date, you prepare, you load-test. Ordinary Tuesdays don’t get that treatment. And that’s exactly where you find out whether a company has a real structure for handling failure, or just a hope that it’ll sort itself out.
An SLA isn’t the “99.9% uptime” line in the contract
When a business owner hears “SLA,” they usually picture one number in a contract. In practice that number guarantees almost nothing on its own. 99.9% a month works out to roughly 43 minutes of downtime you’re technically allowed. The real questions are different: who finds out, how fast, who’s actually on the hook at 3am on a Sunday, and what even counts as an “incident” — a full outage, or also the case where checkout is broken but the catalog loads fine.
A working SLA is a document with several layers: a target availability number, response times by problem type, an emergency contact channel, and a clear line around what support covers and what it doesn’t. Skip that last part and the whole document is just a nice number for the sales page.
What downtime actually costs
This is where most people get the math wrong. Owners multiply average daily revenue by hours down and either panic at the big number, or shrug it off as “it was overnight, barely any traffic.” Neither approach holds up, because losses depend heavily on which specific window you’re talking about.
Something closer to reality: a store with a $45 average order and 40 orders an hour during the evening peak. An hour of downtime from 7 to 8pm isn’t “one hour out of the day” — it’s roughly $1,800 in orders that either went to a competitor or never happened, plus some share of customers who just don’t come back after a bad experience. That same hour at 5am costs an order of magnitude less. Without a rough table of “losses by hour of day,” any conversation about incident priority is happening blind.
Severity tiers: not every bug is a fire
The classic mistake is treating everything with the same urgency. Either the team burns out from constant late-night pages over minor issues, or a genuinely critical outage gets buried under a pile of small tickets. A workable split usually looks like this:
- Critical (P1) — site down, checkout broken, orders not saving. Response measured in minutes, not hours.
- High (P2) — a major feature is broken: search, filters, account pages. The business keeps running, just with friction.
- Medium (P3) — cosmetic or localized bugs that don’t block a purchase.
- Low (P4) — everything else, goes into the regular backlog.
Each tier gets its own contracted response window. A P1 at 2am on a Sunday should page the on-call engineer. A P4 waits until Monday without drama. Once that split is written down ahead of time, the worst part of any incident disappears — the argument about how serious this actually is while customers are already unable to check out.
Monitoring that catches it before your customers do
“The site pings back” isn’t monitoring — it’s a comfort blanket. Real failures usually hide deeper: a job queue that stopped draining and is quietly growing, a database throwing slow queries, a payment provider replying in 8 seconds instead of 200 milliseconds. On paper, everything is “up.”
Which is why it’s worth watching business metrics directly, not just homepage uptime: how many orders came in over the last 15 minutes compared to the same window a week ago, how many payments failed, how much the queue has grown. A sudden drop in successful checkouts is a far earlier and more honest signal than any uptime checker will give you.
Who picks up the phone at 3am
Escalation is where good intentions run into reality. “We’ll message on Slack and someone will answer” isn’t a plan, it’s a hope. A working setup answers three questions in advance: who’s first on call, what happens if they don’t respond within 10 minutes, and who makes the call when a problem crosses team boundaries — say, hosting, the payment gateway, and the codebase all need attention at once.
Without a backup contact in that chain, one vacation or one dead phone battery turns a critical incident into “we’ll deal with it in the morning.”
What support doesn’t cover — and why that needs saying upfront
Most client disputes start here. Site support typically isn’t on the hook for the hosting provider’s own outage, a third-party payment gateway going down, a shipping API failing, or DDoS traffic above a certain threshold — those sit with separate vendors under separate agreements. If that’s not spelled out, the client finds out where the line is exactly during the outage, at the moment emotions are already running highest.
The honest move is handing over a plain list upfront: what’s covered, what isn’t, and who owns each adjacent system. It sounds like a footnote when you’re signing the contract. During an actual outage, that footnote decides how fast anyone starts fixing the problem instead of arguing over whose problem it is.
A runbook: what to do when everything’s on fire
When a server goes down at 2am, nobody wants to be reconstructing the system architecture from memory in that moment. A runbook is a step-by-step document for known failure patterns: payments down, database unreachable, job queue backed up, site throwing 500s everywhere. For each one — where to check logs first, which commands to run, who to pull in if the first step doesn’t fix it.
Without that document, every outage turns into an investigation from scratch, even for people who saw the exact same failure a month earlier.
After it’s back up
This is where most teams drop the ball. The site’s working again, everyone exhales, and that’s the end of it. Worth stopping instead to go through what actually happened, why monitoring didn’t catch it sooner, whether the response could’ve been faster, and what needs to change in the code or the process so the same root cause doesn’t show up again in two months.
A blameless post-mortem isn’t a box-ticking exercise. A team that’s afraid to write down its own mistakes tends to repeat them. A team that walks through an incident calmly ends up with noticeably fewer repeats of the same failure over the following year.
The short version
An SLA on paper doesn’t guarantee anything by itself. What guarantees something is the structure behind it: who’s responsible for what, within how many minutes, with which backup contact, and what happens next once the problem has already started. It’s cheaper to build that structure once than to improvise it mid-outage every single time — and pay for the improvisation in lost orders.
