Incidents are inevitable; chaotic incidents are a choice. The difference between a 12-minute blip and a 4-hour war room is rarely the underlying fault — it is whether the team has a rehearsed process, clear roles, and a learning culture that turns each failure into permanent improvement. This article lays out a working incident-management process and the blameless postmortem discipline that stops the same outage recurring.
The incident lifecycle
Every incident moves through the same stages, and naming them gives the team a shared mental model under pressure:
- Detect — automated monitoring or a customer report surfaces a problem.
- Declare — someone formally opens an incident. This is the single most-skipped step and the most important; an undeclared incident has no owner.
- Triage — assess scope and impact, assign severity, pull in responders.
- Mitigate — stop the bleeding. Restore service by any safe means available.
- Resolve — confirm full recovery and close the incident.
- Learn — run the postmortem and drive the action items to completion.
The order matters. Mitigate before you root-cause. Understanding why the database fell over is a postmortem activity; getting customers back online is the incident.
Severity levels and what they trigger
Severity is not a feeling — it is a defined matrix that determines who gets paged, how fast, and how loudly you communicate. Agree it in advance so nobody is debating definitions at 02:00.
| Sev | Impact | Examples | Response | Comms |
|---|---|---|---|---|
| SEV1 | Critical: major outage or data loss | Checkout down, customer data exposed | Page IC + responders immediately, 24/7 | Exec notify + public status page within 15 min |
| SEV2 | Significant degradation | Elevated error rate, one region down | Page on-call within minutes, business + OOH | Internal stakeholders + status page |
| SEV3 | Minor / partial | Non-critical feature broken, slow but working | Handle in business hours | Internal channel only |
| SEV4 | Negligible | Cosmetic bug, single-user issue | Backlog ticket | None |
The classification is a starting estimate — escalate severity freely as you learn more, and never hesitate to over-declare. A SEV2 you downgrade costs nothing; a SEV1 you under-called costs trust.
Clear roles under pressure
The most common failure mode in incident response is everyone debugging at once and nobody coordinating. Assign explicit roles the moment an incident is declared:
- Incident Commander (IC) — owns coordination, not the keyboard. Decides, delegates, keeps the timeline moving, declares severity changes and resolution. The IC does not fix the problem; they run the response.
- Operations / Subject-matter responders — the people actually investigating and applying mitigations, directed by the IC.
- Communications Lead — owns internal updates and the external status page so the IC is not interrupted every five minutes by "any update?".
- Scribe — maintains a timestamped log of what happened and what was tried. This log is the raw material for the postmortem; reconstructing it from memory afterwards loses critical detail.
In a small incident one person may wear several hats, but the IC role must always be explicit and singular.
On-call and escalation
A humane, effective on-call rota is the foundation. Keep rotations short enough to avoid burnout, ensure every alert is actionable (an alert nobody acts on is noise that trains people to ignore the pager), and define a clear escalation policy: if the primary on-call does not acknowledge within, say, 5 minutes, auto-escalate to secondary, then to the engineering manager. Encode this in your paging tool rather than relying on someone remembering who to call.
# Opsgenie / PagerDuty-style escalation policy (illustrative)
escalation_policy:
name: payments-oncall
rules:
- notify: schedule:payments-primary
escalate_after: 5m # no ack -> next rule
- notify: schedule:payments-secondary
escalate_after: 5m
- notify: user:eng-manager
escalate_after: 10m
Mitigation-first mindset
The instinct of good engineers is to understand the problem fully before acting. In an incident that instinct is a liability. The goal is to stop customer impact as fast as safely possible — roll back the last deploy, fail over to a healthy region, shed load, toggle a feature flag off. Root cause can wait. A rollback that resolves the symptom in two minutes beats a perfect diagnosis delivered an hour later. Capture enough state (logs, metrics, a snapshot) to investigate afterwards, then mitigate.
Communication
Internally, run the incident in a dedicated ChatOps channel where the bot posts state changes, the scribe logs actions, and stakeholders read without interrupting responders. Externally, a status page is non-negotiable for customer-facing services: an honest "we are investigating elevated error rates" posted within 15 minutes buys enormous goodwill compared with silence. Update on a cadence even when there is nothing new — "still investigating, next update in 30 minutes" — and post a clear all-clear at the end.
The blameless postmortem
A postmortem is worthless if people are afraid to be honest in it. Blameless means you treat the people involved as well-intentioned actors who made reasonable decisions given the information and tooling they had. The output is a list of systemic fixes, not a list of who to blame.
Replace "the engineer ran the wrong command" with "the deploy tool allowed an unguarded production command with no confirmation". The first ends the conversation; the second produces an action item.
"Human error" is never a root cause — it is a prompt to ask why the system permitted that error to cause harm. A blameless postmortem template:
- Summary — one paragraph: what happened, impact, duration.
- Impact — users affected, revenue, SLO/error-budget burn, duration of degradation.
- Timeline — timestamped sequence from detection to resolution (straight from the scribe's log).
- Contributing factors — the chain of conditions that combined to cause the incident; almost always plural.
- What went well / what went poorly — including the response itself, not just the fault.
- Action items — each with a named owner, a due date, and a tracking ticket. Categorise as detection, mitigation, or prevention.
- Lessons learned — what the team now understands that it did not before.
Metrics that matter — and their misuse
Track the lifecycle metrics, but understand what they do and do not tell you:
| Metric | Measures | Misuse to avoid |
|---|---|---|
| MTTD (detect) | Monitoring effectiveness | — |
| MTTA (acknowledge) | On-call responsiveness | — |
| MTTR (resolve) | End-to-end recovery | Treating as a target/SLA, or comparing across unlike incidents |
These are aggregate trend indicators, not individual performance scores. The instant you turn MTTR into a target, people game it — closing incidents early or under-declaring severity. Use them to spot trends ("our MTTD is creeping up — is our alerting decaying?"), never to grade engineers.
Building a learning culture
The action items are where postmortems live or die. A postmortem that produces a beautiful document and no completed follow-up is a ritual, not a practice. Review open action items in regular operational reviews, give them the same priority as feature work, and share postmortems widely so the whole organisation learns from each failure. Over time this is what bends the incident curve downward — not heroics, but the disciplined conversion of every outage into a permanent systemic improvement.
A calm, rehearsed incident process and a genuine blameless culture are learnable skills. If you want i2zone to design your incident response, on-call, and postmortem practice, get in touch.