The Inevitability of Failure and Importance of Repair
I’m often amazed at the Sisyphean lengths people will go to try and prevent failure, yet prepare so little for its inevitability. Ed Catmull, president of Pixar, noted the following in his book Creativity Inc.
Do not fall for the illusion that by preventing errors you won’t have errors to fix.
The truth is the cost of preventing errors is often greater than the cost of fixing errors later.
There’s nothing wrong with attempting to prevent failures that are easily preventable. But such preventative measures have to be weighed against the friction and cost the measure introduces. Lost in this calculation is the consideration that much of the energy and effort that goes into prevention might be better spent in preparing to respond to failure and the repair process.
This is a lesson that’s not just true for software, but all aspects of life. The following are examples of where this principle applies to social policy, parenting, relationships, and code.
The “War on Drugs” is a colossal failure of social policy…
If there is one number that embodies the seemingly intractable challenge imposed by the illegal drug trade on the relationship between the United States and Mexico, it is $177.26. That is the retail price, according to Drug Enforcement Administration data, of one gram of pure cocaine from your typical local pusher. That is 74 percent cheaper than it was 30 years ago.
So after thirty years spent, $51 Trillion (yes, Trillion!) spent, not to mention the incredible social costs, the result we get for all that expenditure is it’s 74 percent cheaper for that hit of cocaine. Wall street traders are rejoicing.
It doesn’t take 5 Nobel Prize winning economists to tell you that the drug war is a failure.
The idea that you can tell people to “Just Say No” and that will somehow turn the tide of human nature is laughably ridiculous. This is the failure of the prevention model.
A response that focuses on repair as opposed to all out prevention realizes that you can’t stop people from taking drugs, but you can help with the repair process for those who do get addicted. You can get better results if you treat drugs as a health problem and not a criminal problem. It’s worked very well for Portugal. Ten years after they decriminalized all drugs, drug abuse is down by half.
This development can not only be attributed to decriminalisation but to a confluence of treatment and risk reduction policies.
It’s a sane approach and it works. Locking an addict up in jail doesn’t help them to repair.
A while back I wrote about the practice of Reflective Parenting. In that post, I wrote about the concept of repairing.
Now this last point is the most important lesson. Parents, we are going to fuck up. We’re going to do it royally. Accept it. Forgive yourself. And then repair the situation.
If there’s ever a situation that will disabuse a person of the notion that they’re infallible, it’s becoming a parent. An essential part of being human is that mistakes will be made. Learning how to gracefully repair relationships afterwards helps lessen the long term impact of such mistakes.
Perhaps I’m fortunate that I get a lot of practice fucking up and then repairing with my own kids. Just the other day I was frazzled trying to get the kids ready for some birthday party. I told my son to fill out the birthday card, but avoid a splotch of water on the table while I went to grab a towel. Sure enough, he put the card on the water. I was pissed. I berated him for doing the one thing I just finished explicitly telling him not to do that. Why would he do that?! Why didn’t he listen?!
His normal smile was replaced with a crestfallen face as his eyes teared up. That struck me. When I calmed down, he pointed to a little plate full of water on the table. He thought I had meant that water. “Asshole” doesn’t even begin to describe how much of a schmuck I felt at that moment. It was a total misunderstanding. He didn’t even see the splotch of water next to the more conspicuous plate of water.
I got down to his eye level and told him that I’m sorry. I made a mistake. I understand how my instructions would be confusing. I was sincere, remorseful, and honest. We hugged it out and things were fine afterwards. Learning to repair is essential to good parenting.
I’ve been reading Difficult Conversations: How to Discuss What Matters Most. This book is a phenomenal guide to communicating well, both at work and home. Even if you think you are great at communicating with others, there’s probably something in here for you.
It helped me through a recent difficult situation where I hurt someone’s feelings. I had no idea that my words would prompt the response it did and I was surprised by the reaction. Prior to reading this book, my typical approach would be to try and defend my actions and help this person see the obvious reason in my position. I would try to win the argument.
Difficult Conversations proposes a third approach, rather than try to win the argument, it suggests you move towards a learning conversation.
Instead of wanting to persuade and get your way, you want to understand what has happened from the other person’s point of view, explain your point of view, share and understand feelings, and work together to figure out a way to manage the problem going forward. In so doing, you make it more likely that the other person will be open to being persuaded, and that you will learn something that significantly changes the way you understand the problem. Changing our stance means inviting the other person into the conversation with us, to help us figure things out. If we’re going to achieve our purposes, we have lots we need to learn from them and lots they need to learn from us. We need to have a learning conversation.
What I’ve learned is that people in general aren’t irrational. They only appear to be irrational because you are often missing a piece of context about how they view the world and interpret the actions of others.
This becomes crystal clear when you consider how you interpret your own actions. When was the last time you concluded that you acted with malicious intent or irrationally? How is it that you always act rationally with good intent, and others don’t? Given your impeccable track record, how is it that sometimes, others ascribe malice to your actions? Well they must be irrational! Or is it that they are missing a piece of context that you have? Could it be possible, when you’ve been on the other end, that you ascribed malice in a situation where you really were missing some information?
It’s not until you realize most people are alike in this way that you can start to have more productive learning conversations - even with folks you strongly disagree with.
Back to the story, despite all my good intentions and all my efforts to be respectful, I still failed and hurt my friend’s feelings. It’s just not possible to avoid this in every situation, though I strive to greatly reduce the occurrences. Fortunately, I’ve prepared for failure. By focusing on a learning conversation, we were able to repair the relationship. I believe it’s even stronger as a result.
There’s so many examples in software, it’s hard to point to just one. So I’ll pick two. First, let’s talk about The Thing About Git. I’ve linked to this post many times because one of its key points really resonates with me.
Git means never having to say, “you should have”
If you took The Tangled Working Copy Problem to the mailing lists of each of the VCS’s and solicited proposals for how best to untangle it, I think it’s safe to say that most of the solutions would be of the form: “You should have XXX before YYY.” … More simply, the phrase: “you should have,” ought to set off alarm bells. These are precisely the types of problems I want my VCS to solve, not throw back in my face with rules for how to structure workflow the next time.
Git recognizes that people make mistakes and rather than tell you that your only recourse is to grab a time machine and do what you should have done in the first place, it gives you tools to repair mistakes.
The theme of preparing for failure applies just as much to software and systems as it does to dealing with people.
There are a lot of backup systems out there. And to a degree, backups are a step in recognizing the value in preparing for disasters. But as any good system administrators know, backups are not the important part of the process. Backups are nothing without restores. Restores are what we really care about. That’s the “repair” step when a hard-drive fails.
Systems and policies that require 100% failure prevention to work are highly suspect. Such a system should trigger your Spidey sense. When building a system or policy, think not only about how the system or policy might fail, but how users of the system and those subject to the policy might fail. And give them tools to recover and repair failures. Perhaps the only guarantee you can provide is there will be failure. So prepare for it and prepare to recover from it.