Quantum Thinking

Why good product managers falsify their assumptions

Yesterday I watched my four year old son act out a scene from his favourite TV show, PAW Patrol.

I can’t help but smile when I see him play multiple characters at once. He’ll happily play judge, jury and executioner at the same time. Changing his tone of voice, body language and positions on his ‘stage’. His poor younger brother usually gets relegated to a static prop at the side of the room.

This seems to be a fairly intuitive trait that kids possess — the ability to literally juggle multiple personas in their head at the same time. They can watch and imitate different people surprisingly well. Imagination and mimicry are amazing tools.

As we grow older, that ability to balance different perspectives in our head seems to diminish. Unless you’re a paid actor, you probably don’t spend much time acting out different personalities. As an adult, the reality is that our conscious mind calls the shots and the ego likes to be proven right. We’ll actively drown out opposing views to help confirm what we already believe. We need to be the lead character in our own production and that tends to be a solo performance.

Yet when it comes to product development, this same attitude can prove totally destructive. Confirmation bias is the ultimate potential killer. If you don’t look for other viewpoints and contradictory opinions, you’ll always be susceptible to the filter bubble.

I remember chatting with the Head of Research in Spotify over lunch one day. At the time he was responsible for hundreds of user researchers and data scientists across the company. He told me that if had a magic wand and could change one thing about how product teams operate, it would be to remove the validation trap. Time and time again, he'd seen high potential projects fall short because the product manager had 'latched' onto a specific product implementation. They would then spend the next few months 'validating' their great idea. Ultimately though, the market doesn't care about what anyone thinks. Conducting primary research to confirm an idea might seem like a good use of time. But it’s still self-referential design. It’s just hidden under the facade of scientific rigour. Real scientists don't validate.

Design like you're right, test like you're wrong

At Product Buffs, we don’t teach people how to validate their ideas. We teach them how to falsify assumptions. The reason is pretty simple — any scientific law is conclusively falsifiable, it's not conclusively verifiable. Karl Popper is known as the originator of this principle (the 'Falsification Principle'). He believed that the scientific method was founded on the idea of falsifiability or testability. In short, a theory isn't scientific unless it could be proven false. Science aims to be disproven, not confirmed.

A famous example is the Black Swan analogy that Nassim Nicholas Taleb outlined in this article:

Before the discovery of Australia, people in the old world were convinced that all swans were white, an unassailable belief as it seemed completely confirmed by empirical evidence... It illustrates a severe limitation to our learning from observations or experience and the fragility of our knowledge. One single observation can invalidate a general statement derived from millennia of confirmatory sightings of millions of white swans. All you need is one single (and, I am told, quite ugly) black bird.

In my experience, the best product managers are able to demonstrate quantum thinking. This means that they can hold an optimistic vision in their head while actively trying to find evidence that could disprove their product hypotheses. They don’t assume that all swans are white. They try to find the black swans. They want to find the hidden truths, even if these crush their prior beliefs. They look for disconfirming information to expand their perspective. They don’t try to validate.

I still struggle with this all the time. It's just human nature to want to prove that you're right. I was lucky to be trained as a lawyer when I left university. One of the ways that lawyers approach a case is to think about how the other side will argue their case. You don't want the opposition to find holes in your brief in front of a packed courtroom. So it’s good practice to find these weaknesses in your argument early on. You need to see the upside and the downside concurrently and use that to your advantage.

On one of my first cases as a junior lawyer, I remember doing an exercise with a seasoned partner of the firm. We outlined our strategy and then spent hours picking it apart and arguing the case from the opposition's perspective. Afterwards, I genuinely thought that our case was going to be thrown out! Thankfully the lawyer with 20+ years experience was able to talk me off the ledge. She explained that having doubts was all part of the process. We still had a great case. It just made sense to highlight any potential pitfalls in advance. We went on to win our case. I still think about that exercise when I'm doing research today.

Quantum thinking is a very tricky thing to get right. If you're too pessimistic, the project will never get off the ground. If you're too optimistic then you'll miss clear signals along the way — even when you're heading towards a complete dead end. The same conflict applies for every startup that wants to get traction. Founders need to be both optimistic and paranoid. That's a weird and paradoxical combination of emotions to bundle together. If only the paranoid survive, what does that mean for product development?

Patrick Collison from Stripe sums it up well in a previous interview:

“I personally am a perennial optimist; a paranoid optimist. I have visions of everything going wrong but I’m an optimist in the end. Most people in the world are relatively sceptical; they are not wrong — most things don’t work. Most people are correctly sceptical. So how do you cultivate unusual optimism? You need to foster a sense an ethos, even given all the iterations, and [ask yourself]: where can this go?”

I think there are systematic ways you can deal with these conflicting cognitive biases when building new product. For example, you could take the Vision Brief as the optimistic side of the argument. This helps you answer the question about where any potential project could go. You then need to invest in your foundational research so that you can go out and falsify your hypotheses. This part makes up the pessimistic side of your argument. If you're doing it right then you're basically balancing peril and promise together in your mind at the same time. It’s mental tightrope walking.

I recently went through this exact exercise with ARRO, the product concept I'm exploring. I had a hypotheses that including a life coach as part of the offering would help keep people accountable and improve the overall experience. So coaching became part of the original product scope (see below).

I then set out to falsify this hypothesis. To do that I set up a Roam document and copied a lot of the functionality for the proposed ARRO build (eg goal setting, weekly planning, daily journal etc). I then spoke with a qualified coach who kindly offered to test it out with a group of us. We shared access to the Roam document and the coach was able to follow along as we updated our inputs everyday (example daily journal template below). The coach would then chime in with encouragement or prompts.

My hunch was that I'd be more likely to complete a goal if I had both peers and a coach helping to steer me towards a particular outcome. Instead, I found the coaching piece unhelpful. It didn’t really drive any additional output. Although the coach was responsive and professional in how he tried to nudge me along, it felt like a chore for the two of us. I much preferred the organic peer feedback from the group. It didn't feel as cumbersome and the group accountability was more effective at helping me reach my goals.

So the coaching hypotheses was falsified and I've dropped it from the MVP scope. It’s a clear example of an idea that's been disproven. I was delighted to learn that now — not after I'd baked the feature into the product and learnt the hard way. Measure twice, cut once.

I think Steve Jobs called it perfectly when he talked about what focus really means. It's more often a case of deciding what not to build instead of what you could build. Stripping a product down to its core essence and throwing away everything else.

People think focus means saying yes to the thing you’ve got to focus on. But that’s not what it means at all. It means saying no to the hundred other good ideas that there are. You have to pick carefully. I’m actually as proud of the things we haven’t done as the things I have done. Innovation is saying ‘no’ to 1,000 things.

So try to say no to your product hypotheses more often than you say yes. Play judge, jury and executioner at the same time. Take the time to find the counter argument and don't be afraid to shelve an idea if your instincts are proven wrong. In the short term this might hurt your ego. It will hurt a lot less though than building something that nobody wants.

If you can master the skill of quantum thinking when doing product discovery, it will set you apart from the pack. It will also drastically improve the quality of the products you ship.

Just remember to falsify, don't validate.