A Production Outage During a Traffic Spike: 55 Minutes to Restore Service

A marketing campaign went live and the traffic it drove arrived faster than the application could absorb it. The app did not fail cleanly. It went partially down, which is the harder version: some traffic served, some failing, and no single obvious break to point at. [NEEDS INPUT: which application and traffic profile, including the campaign channel and the peak request volume it drove] The campaign was live and driving traffic the application was failing to serve, and the campaign did not pause because the platform was struggling. My team owned the platform. The clock was not ours to negotiate with.

Partial failure is the worst signal to read because the system keeps reporting that it is up. Health checks pass. Dashboards stay green in aggregate while a share of real users get nothing. What told me it was real was the correlation: degradation started when the campaign started. A system that has been stable and then breaks the moment traffic changes is not suffering a coincidence, the load is the cause until proven otherwise.

In an outage the first question is not what broke, it is what is still working and who owns the next hour. I split the failing surface from the parts still serving, and separated traffic the business actually needed during a campaign from traffic that could wait. That ordering matters: you cannot protect a critical path you have not yet named.

A release landing near the campaign start is the usual explanation, so that is what I checked first. I ruled it out because the degradation tracked traffic volume rather than release time, worsening as load rose and easing as it fell, which a bad build does not do. I also ruled out chasing the root cause before restoring service. Diagnosing under load means shipping an untested change into a system already failing, and the time to a confident diagnosis is reliably longer than the time to shed load. Root cause is an after-action task. Availability is the incident.

I chose to restore the critical path by degrading everything else: route non-critical traffic away and turn on temporary safeguards, rather than hold full functionality while hunting the cause. What that cost is worth stating plainly. I deliberately made parts of the product worse for every user in order to keep the path the campaign depended on answering. Some traffic was not served on purpose. The safeguards bought availability with configuration that was never meant to survive the incident, and they left the underlying cause unproven at the moment service came back. I also spent engineering attention on a 30-minute client update cadence, which is time not spent on the fix.

Within ten minutes there was an incident channel and named owners, assigned individually rather than to a group thread, because ambiguity in the first ten minutes is what turns a partial outage into a long one. Non-critical traffic was routed away and the temporary safeguards went on. In parallel I ran the client side myself on a 30-minute cadence: what we knew, what we had changed, what the next update would contain. Fixed intervals matter more than good news. A client who knows the next update arrives in thirty minutes stops asking every five, and the engineers stay on the incident instead of the inbox.

Service was restored within 55 minutes. The second thing I count is the cadence itself. The client had an update every thirty minutes — what we knew, what we had changed, what the next update would contain — rather than a silence they had to break. An outage becomes a trust problem when the customer learns about it from their own users. [NEEDS INPUT: measured campaign impact during the outage window — conversions, orders, or revenue]

The ten-minute rule is the one I hold to: an incident is not open until there is a channel and a named owner per workstream, and that happens before anyone opens a log. I also moved traffic classification earlier — deciding which paths are critical during an outage is deciding it too late — and I run a fixed 30-minute update cadence on a live incident whether or not the client has asked for one.