A Production Outage During a Traffic Spike: Moved to a Pre-Built AWS Backup in 30 Minutes

A quiz and olympiad platform's brand recognition on Facebook was already high, and a campaign there drove traffic the client had sized for 8,000 concurrent users. It reached 20,000. Login and registration started failing: specifically, the step after payment where a user's token comes back and they return to the platform began timing out.

The failure sat in one place: the return trip after payment, not the payment itself. That is a latency signature, not a capacity one: a server that is out of compute fails broadly, a server that is waiting too long on one round trip fails at exactly the step that round trip belongs to.

The database was hosted in Tokyo, the default region on Supabase; Singapore was never separately considered or set up as an alternative before this happened. That was the actual gap: region was never a decision anyone made, it was just what defaulted in. The users and the application server were in Bangladesh. Under normal load the extra distance cost milliseconds nobody noticed. Under 20,000 concurrent users, every one of those round trips queued behind the last, and the token-return step (already the most latency-sensitive point in the flow) was where it broke first.

I did not try to tune the existing setup under live load with users still arriving. An AWS ECS and EC2 backup had already been built for this kind of situation, before this traffic spike happened, and the decision was simply to use it rather than attempt a fix on infrastructure that was actively failing under a load it was never sized for.

Moving the platform to the AWS backup took about 30 minutes. An SSL verification issue slowed the cutover partway through, and the work started around three in the morning, while the campaign traffic was still live. Roughly 2,000 registrations failed to complete during the spike before the move finished.

The move to the AWS backup took about 30 minutes from the point I started it. About 2,000 registrations failed during the spike itself; support recovered them afterward by hand through the admin panel, crediting the accounts affected. The cross-region latency problem itself was fully resolved by the move: it did not recur afterward.

The backup that mattered was built before the traffic spike, not during it. An AWS ECS and EC2 environment existed and was ready to receive traffic because someone had already asked what happens if the primary setup cannot hold, before it could not. That is the same pattern as a payment gateway outage I have also documented: the response that looks fast during an incident is almost always preparation that happened earlier, not improvisation in the moment.