The Danger of Optimism

I am an optimist and I always have been. I can find a silver-lining in anything. However, I’ve started to realize that Optimism can be a very dangerous attribute for a software professional. Now before you think I’m going to start preaching pessimism, let me assure you that I think that’s even more dangerous. What you need is Rational Optimism. Just what do I mean by that? Let’s start with a story highlighted by Jim Collins in his book Good to Great.

In his book, Collins tells the story of Admiral Jim Stockdale, the highest ranking United States military officer in the “Hanoi Hilton” prisoner-of-war camp in the Vietnam War. In his eight years in prison, ending in 1973, he was tortured over 20 times. Collins quotes Stockdale describing his strategy for dealing with this gruelling tenure:

I never lost faith in the end of the story, I never doubted not only that I would get out, but also that I would prevail in the end and turn the experience into the defining event of my life, which, in retrospect, I would not trade.

Collins then asked Stockdale about those who didn’t make it out. He gave this surprising answer:

Oh, that’s easy, the optimists. Oh, they were the ones who said, ‘We’re going to be out by Christmas.’ And Christmas would come, and Christmas would go. Then they’d say, ‘We’re going to be out by Easter.’ And Easter would come, and Easter would go. And then Thanksgiving, and then it would be Christmas again. And they died of a broken heart.

Collins brought these ideas together into what he called the “Stockdale Paradox”:

You must never confuse faith that you will prevail in the end – which you can never afford to lose – with the discipline to confront the most brutal facts of your current reality, whatever they might be. (Collins, p. 83)

In the software world, I have often heard this summarized with the much shorter phrase “Hope is not a strategy.” Too often in software development I have heard teams tell me they think they can get something done on time. Unfortunately, I completely agree with them. I think they can too. They are brilliant coders who are willing to work sometimes crazy hours to see a project through.

It makes me happy to think that we will get everything done. That’s what optimists do: come up with perfectly reasonable and logical reasons why we should be happy in any given situation. Unfortunately, one of the problems with optimists is that we lack the imagination to properly asses the unseen, and often unpredictable, risks to our projects. It might surprise you to learn that software development is hardly alone in this dilemma. In her recent book Bright-sided: How the Relentless Promotion of Positive Thinking Has Undermined America Barbara Ehrenreich discusses example after example showing how irrational optimism has lead to the downfall of the stock market, American businesses, political parties, religions, and personal lives. While there are key points in her book I disagree with, the central premise is strongly applicable to software development.

How do you avoid letting your optimistic tendencies lead you down a primrose path where nothing ever goes wrong? Here are some of the tools I have used.

Find a Pessimist. I like to have at least one pessimist on every team. This isn’t someone who feels that everything is doomed to failure – hopefully you refused to hire that person. Rather, this is someone who is cursed with the ability to vividly imagine all the things that might go wrong, and who isn’t afraid to bring them up. Give this person plenty of opportunity to remind everyone what could happen if proper planning isn’t performed.

Remember the Past. We’ve all been part of projects that failed to finish on time, or even at all. The optimist will quickly dismiss those experiences as bad luck. You’ll probably even remind yourself that they were valuable learning experiences (there’s that silver lining again). However, they can only be learning experiences if you honestly believe they could happen again. Remind yourself of this every time you are tempted to start listing all the positive reasons why your hopes are going to come true.

Brainstorm the Risks. While optimists aren’t often very imaginative when it comes to envisioning possible risks, there is a way to overcome that natural weakness. In his book Six Thinking Hats Edward De Bono describes the power of directed brainstorming. Even a group of crazed optimists can come up with an impressive list of project risks if that’s the only topic they are allowed to brainstorm about. The act of focusing on one particular kind of idea makes it easier to come up with new ideas that are similar. If you are afraid that you’ve got too many optimists and you aren’t looking at a problem realistically enough, get your team together in a room and brainstorm only the risks to your project.

Measure Past Performance. If you have enough similar projects to draw upon you can start predicting future performance by looking at the past. This is the basis for Story Point based estimating (see Mike Cohn’s book Agile Estimating and Planning or Jack Milunsky’s blog post on the topic). If you have collected metrics on past projects you can predict what your team’s average velocity will be on new, similar, projects. This velocity takes most forms of risk already into account. If your optimistic estimates don’t match what your past velocity and metrics are telling you, then you’ve fallen victim to your own nature.

Despite all of the challenges that optimists face in the software profession, I firmly believe they are still the right kind of people to have in charge of your projects. Very few revolutionary projects were ever lead by pessimists. I can’t think of a single market-changing business that didn’t have an optimist at the helm. But if you are an incurable optimist (like I am) you need to make sure you use proper tools to keep your optimism in check. If you can do that, you’ll get all the amazing benefits of optimistic thinking, without suffering the potentially disastrous consequences it might otherwise bring.

What tools do you use to keep your optimism in check? Can you think of experiences where irrational optimism helped a project, rather than hurt it?


3 thoughts on “The Danger of Optimism

  1. Hi, Cliff. Rob Wylie pointed me here. I have to admit I’ve read only your most recent post, but it did strike a few chords.

    Perhaps the only real difference between a world of optimists and a world of pessimists is that the road to Hell would be a lot more cheerful with the optimists (although it might be a shorter one too).

    I spent a significant part of my life as a mechanical engineer designing a product that, if it were to fail, would be sure to kill at least one person every time. This knowledge served to keep my attention on the potential failure modes every step of the way. This experience served me well when the engineering design world took up the “right first time” mantra in the 1980s.

    The software world seems to be rather different. The strategy of Microsoft et al appears to be “get it to market quickly even though it doesn’t quite work, then we can get the users to tell us what’s wrong and sell them another version 18 months later”. It’s fortunate that Boeing and Airbus don’t design their products like that, although it seems that an increasing number of consumer product manufacturers are starting to follow the MS example. But I digress…

    My “failure mode” training became second nature to the extent that I automatically ask “what if…” questions at almost every decision point I encounter. This causes the optimist to brand me as a pessimist, whereas I see myself as a realist. When one person refers to another as either a “glass half full” or a “glass half empty” person, I point out that the engineer’s response is that the glass is too big.



  2. Pingback: Estimation and the Goodreads Reading Challenge | Conjectures and Refutations

Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out /  Change )

Google photo

You are commenting using your Google account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s