The phrase “never let a good incident go to waste” might seem cynical at first. However, in the world of technology, operations, and business in general, it holds a deeper wisdom. It’s a call to use setbacks, failures, and incidents as catalysts for improvement. At the heart of this philosophy lies the postmortem – a critical process vital for turning incidents into growth opportunities.

What is a Postmortem?

A postmortem (or a post-incident review) is a structured analysis conducted after an incident has occurred. This can encompass anything from a minor glitch to a critical system outage. The primary goal of a postmortem is to:

  • Understand: Thoroughly understand what happened, why it happened, and its full impact.
  • Learn: Extract lessons that prevent recurrence and improve overall system resilience.
  • Improve: Implement changes to processes, technologies, or training to make the system better equipped to handle similar events in the future.

Why “Never Let a Good Incident Go to Waste”?

Here’s why embracing this philosophy is essential:

  1. Incidents are inevitable: No matter how well-designed your system is, incidents will happen. It’s the nature of complex systems, especially in the ever-evolving world of technology.
  2. Learning opportunities: Incidents, while disruptive, are incredibly rich learning opportunities. They expose weaknesses, vulnerabilities, and hidden dependencies that routine operations usually don’t reveal.
  3. Prevention is key: The knowledge gained through postmortems allows organizations to proactively address issues rather than continually reacting to them. This helps to minimize the risk of more severe incidents in the future.
  4. Building resilience: Analyzing incidents fosters a culture that embraces learning and continuous improvement. It helps build antifragility – the ability of a system to not just withstand shocks but grow stronger from them.

The Postmortem Process

Effective postmortems follow a structured approach. Here’s a breakdown of the key phases:

  1. Preparation:

    • Postmortem template: Have a pre-defined template to ensure consistent, detailed documentation.
    • Blameless culture: Emphasize the importance of a “blameless” environment where the focus is on learning, not assigning fault.

  2. Data gathering:

    • Timeline: Construct a detailed timeline of events leading up to, during, and after the incident, including actions taken.
    • Logs and metrics: Collect relevant logs, system metrics, and any monitoring data.
    • Customer impact: Document the impact on customers or users, both qualitatively and quantitatively.

  3. Root cause analysis:

    • The “5 Whys”: Use techniques like the “5 Whys” to dig into the underlying causes, not just the immediate symptoms.
    • Contributing factors: Identify all factors that played a role, including technical, process-based, and human elements.

  4. Recommendations:

    • Specific and actionable: Develop clear, concrete recommendations for reducing the likelihood and impact of similar incidents.
    • Prioritization: Prioritize recommendations based on potential impact, resource requirements, and feasibility.

  5. Follow-through:

    • Ownership: Assign clear ownership for implementing the agreed-upon improvements.
    • Tracking and reporting: Track progress and report on the implementation of postmortem action items.

  6. Sharing and Knowledge Building:

    • Knowledge base: Document and share postmortems internally to build a repository of institutional knowledge.
    • Industry sharing: Consider sharing insights (anonymized if needed) with the wider community to help others learn and improve.

Best Practices for Effective Postmortems

  • Timeliness: Initiate the postmortem process as soon as possible after the incident while memories are still fresh and critical data is easily accessible.
  • Diverse participation: Involve people from different teams who have varied perspectives on the incident. This includes engineers, support staff, product managers, and even those outside the immediate technical realm.
  • Facilitation: Designate a skilled facilitator to keep the discussion focused, constructive, and respectful.
  • External perspective: Consider inviting an external expert or a fresh set of eyes to avoid biases and provide an objective view.
  • Customer focus: Keep the customer impact at the forefront of the discussion. How can you improve the customer experience during and after potential future incidents?

The Importance of a Blameless Culture

One of the most critical aspects of an effective postmortem is a blameless culture. Here’s why it matters:

  • Psychological safety: People are more likely to be honest and open about mistakes if they fear no repercussions.
  • Focus on solutions: A blameless environment shifts the focus towards finding solutions instead of finding scapegoats.
  • Collective learning: When people feel safe to share, the learning potential of a postmortem increases significantly.

Beyond Technical Root Causes

While technical root causes are essential to identify, remember that incidents often have non-technical contributing factors. Postmortems should explore these too:

  • Process gaps: Did any breakdowns in processes, communication, or coordination contribute to the incident?
  • Documentation: Was up-to-date documentation lacking? Would better documentation have helped mitigate the problem?
  • Training: Did the incident highlight areas where additional training or skill development could enhance the team’s response?
  • Decision-making: Were there decision points where different choices might have changed the incident’s outcome?

The Power of Storytelling

Postmortems shouldn’t be dry technical reports. Using a narrative style with storytelling techniques can make the learnings more engaging and memorable:

  • The hook: Begin with a brief, attention-grabbing account of the incident and its impact.
  • The narrative: Structure the postmortem as a journey of discovery, highlighting key questions, challenges, and ‘aha’ moments.
  • Visual aids: Use diagrams, timelines, or screenshots to support the narrative and make insights clearer.

Making Postmortems a Habit

To fully realize the “never let a good incident go to waste” mindset, postmortems need to become a cultural norm. Here are ways to achieve this:

  • Celebrate learning: Recognize and reward teams who conduct thorough postmortems and take proactive steps towards improvement.
  • Non-punitive: Reiterate the blameless nature of postmortems and consistently uphold this commitment.
  • Automate: Make it easy to start a postmortem and document findings with ready-to-use tools and templates.
  • Regular reviews: Incorporate periodic reviews of past postmortems to ensure that lessons are being consistently applied.

Conclusion

Adopting a learning-focused approach to incidents transforms them from sources of stress and frustration into powerful opportunities for growth. By embracing postmortems, organizations foster resilience, continuous improvement, and a culture that values both success and the lessons learned along the way. Remember, it’s not about the incidents themselves, but about how you respond to them that truly defines your organization’s capabilities.