Understanding what Went Wrong

By doing so, you'll learn a great deal about your product, your business, and the way in which you respond to a crisis

Gather all the clues

Before you think about explaining things to your customers, you need to better understand what went wrong yourself. If you followed our advice on Monitoring and Team Communication you’ll already have a great set of information to help identify how things unfolded.

Infrastructure and Processes

Even though it may appear to have been a part of the product which failed, take this opportunity to also look at your business processes, and the way in which you handled the crisis, as you’ve learned to throughout this guide. This can be even more important than the way in which you solve the physical failure.

There is no single cause

So often you’ll find yourself pinning the blame on a single element, such as ‘the database server failed,’ but this is seldom the true root cause. Why did the database fail? Why was there no fail-over in place? Why did it take so long to restore the backups? Why didn’t Bob answer his pager?

There are no ‘causes’ of a failure, only ‘contributors.’ Take your time to properly understand all of the events which put you in a position where your service failed.

"For complex systems there is a myth that deserves to be busted, and that is the assumption that for outages and accidents, there is a single unifying event."

John Allspaw at Etsy

Mitigate the contributing factors

There’s no sure-fire way to prevent things from going wrong again, but you can take reasonable steps to mitigate some, if not most, of the contributing factors. It’s important that you do this; not only for your own peace of mind, but this is the key aspect which will begin to rebuild trust with your customers.

Take Aways

  • Look for a chain of events.
  • Consider your response as well as the issue itself.
  • Make changes to prevent things happening again.

Next Chapter >

Writing a Post-mortem

Did this answer your question? Thanks for the feedback There was a problem submitting your feedback. Please try again later.

Still need help? Contact Us Contact Us