Most teams already know how to run a post-incident review in the mechanical sense: gather everyone involved once service is back, walk the timeline, write it down. The mechanics are not where these fail. They fail on purpose — on what the review is for — and quietly enough that a team can hold them for a year without noticing that nothing downstream changes.
The failure mode is an account, not a change
A post-incident review is supposed to produce a change to the system. What it usually produces is an account of the incident, written for an audience that was anxious while it was happening.
An account is finished when it is coherent — when the timeline explains itself and the response looks defensible in hindsight. A change is finished when something in production or in the process is materially different, and it is graded months later by whether that difference is still there.
In the room the two are indistinguishable. The tell arrives later, when the system is exactly as it was and the same failure class is still open.
A review that cannot point at something that persisted did not happen. It was written.
Restore first, understand second
This ordering is a discipline, not an instinct. Instinct argues the other way: a live broken system is the richest evidence you will get, and rolling it back destroys what you most want to study.
An ECS deployment that failed silently is the cleanest case I have. The deploy completed without raising an error, the container never came up, and the service was down while the pipeline reported success. CloudWatch logs gave me a misconfigured task definition and missing environment variables, and I rolled back to the last stable task revision before fixing anything.
I run delivery, not deploys. The engineering background is what lets me read the logs before making the call rather than waiting to be told what they say — which is why it sits inside a project manager's job rather than beside it.
That rollback cost the live failure environment — the only place to observe the fault in its natural state — and the release the deployment was meant to carry. I accepted both: the logs I held were enough to work from, and every minute spent validating a fix in production is a minute of real outage. Service was restored in 35 minutes; the task definition and IAM corrections landed afterward, on a stable system, behind health check validation.
The precondition is narrow: capture enough before you discard the state. Rolling back without diagnosis is not discipline, it is deferral — you redeploy into the same wall later.
The output is a change to the system, not a document
Four of the incidents I have written up as case studies each ended in something that outlived the review.
A database that stopped answering at peak traffic traced to slow queries and lock contention, not capacity. The fix left indexes and a caching layer in the architecture, so peak load stopped reaching the contention threshold at all — an 80% query performance improvement, and a shape change that removed the condition.
A production outage produced an emergency incident channel with named owners and a fixed stakeholder status cadence. Service came back in 55 minutes; what persisted was the coordination pattern, not the fix. [NEEDS INPUT: what owner-assignment and stakeholder-update intervals were actually used during the 55-minute production outage?]
The ECS failure produced a deployment checklist and automated health checks. A deploy that cannot prove the service is serving no longer counts as successful.
A defect rate climbing across three consecutive sprints produced a root-cause review on test coverage gaps and unclear acceptance criteria, then a stricter Definition of Done and pre-merge quality gates.
Two changed the architecture, two changed the process. Both count. What does not count is a resolution phrased as an intention — be more careful with deploys, write more tests — because an intention has no enforcement surface and decays at the rate attention does.
Blamelessness is a mechanism, not a courtesy
The argument for blamelessness has nothing to do with kindness. The person who understands the failure best is almost always the one whose action sat closest to it. If attribution is expensive, that person narrates defensively and you lose resolution on the only account that matters.
The mechanism is a scope rule: the review examines the system that made the action reasonable, not the action. The question is what made this the sensible thing to do at the time, not why did you do it.
That reframe is what unlocked the ECS case. A deployment that completes without objection while the service is down is a monitoring failure before it is a configuration failure. Nobody arrives at that sentence while defending a deploy.
Blamelessness is not the absence of accountability. Owners are still named — they own the change rather than the fault.
Who should be in the room
Three roles, and the third is the one teams skip: whoever touched the system during the incident, whoever will own the resulting change, and someone with authority to approve it. Without the third, the output is a recommendation, and recommendations are where reviews expire.
Everyone who was anxious during the incident is not on that list. The communication cadence is what keeps them off it: during that outage the client stayed on a fixed update cycle throughout, so nobody arrived afterward needing to be informed. If the review is the first time someone hears the story, it becomes a briefing and the change never gets designed. [NEEDS INPUT: is there a fixed window between restoration and the review being convened, or is it scheduled case by case?]
When a full review is not worth running
A review costs senior attention, the scarcest thing on a delivery team. Skip it when the cause is understood and the fix already permanent, when the issue recurs but has a tracked owner and a date, and when nobody available can authorize a change.
Run one when the incident surprised you. Surprise is a better trigger than severity, because it marks the boundary of what your model of the system covers. The defect-rate case makes that argument: nothing went down, no alert fired, just a number climbing across three sprints — severity effectively zero, and the review still worth the hour. [NEEDS INPUT: is there a written threshold that decides whether a full review runs, or is that call made per incident?]
The reviews worth holding are the ones you can audit later by looking at the system rather than the folder.