Kill It with Fire

A former colleague of mine and an experienced engineer from Google used to like to say, “Anything over four nines is basically a lie.” The more nines you are trying to guarantee, the more risk-averse engineering teams will become, and the more they will avoid necessary improvements. Remember, to get five nines or more, they have only seconds to respond to incidents. That’s a lot of pressure. SLAs/ SLOs are valuable because they give people a budget for failure. When organizations stop aiming for perfection and accept that all systems will occasionally fail, they stop letting their technology rot for fear of change and invest in responding faster to failure. That’s the idea anyway. Some organizations can’t be talked out of wanting five or even six nines of availability. In those cases, mean time to recovery (MTTR) is a more useful statistic to push than reliability. MTTR tracks how long it takes the organization to recover from failure.