How to win at hypothesis testing
You should expect that the first time you attempt to create something truly innovative, something that has never been done before, you will make mistakes and you fail to realize the innovation’s true potential.
A wise man said, quitters never win and winners never quit, but if you don’t win and you don’t quit, you’ve got a gambling problem.
As a responsible business leader, you must develop the delicate sense of when to move on, and that only comes with practice. There is an epidemic of over-commitment of resources to failing projects, called zombie projects because they won’t die. This financial rigour mortis has doomed many commercial ventures. The same is true of projects where market conditions change after launch and late adjustments muddle the results.
Given the new normal of hyper-competition and economic uncertainty across the globe, firms have a better chance of survival using the hypothesis approach from the start. It creates a framework where companies can make serious mistakes as early and as cheaply as possible.
One of the most important practices that distinguish a company as more successful at innovation is the sheer number of attempts that are fully tested. There’s an entire website devoted to the Google Graveyard, which catalogues all the projects the search giant has launched and killed recently. The principle of hypothesis-based testing is sometimes expressed as “Fail fast, fail cheap. Learn faster, learn cheaper.” In this case, “failure” is just a rhetorical device referring to the validation or refutation of hypotheses.
The Hypothesis-based approach to innovation consists of four steps:
- Step one, formulate your assumption as a testable hypothesis, such as customers love cute animal avatars.
- Step two, design and run experiments to test the hypothesis, by introducing a variety of cute animal avatars to representative customers and partners.
- Step three, observe their reactions and act like an anthropologist, following the techniques described in Episode two. Measure results and come to a consensus on what those results indicate.
- Step four, verify or discard hypothesis and return to step one.
These don’t have to be fancy experiments costing thousands of dollars over six months with testing software. It can be as simple as running concepts past selected customers in coffee shops. As validated concept evolve into prototypes and minimal viable products, your testing can become more rigorous and extensive.
All of this may sound eminently reasonable, yet it is rarely practiced in the business world. Instead, the normal procedure is that executives assign project teams that become responsible for innovation. The leaders then follow-up with a solution-based approach, where they define a problem and immediately try to come up with wholly-formed solutions. These teams don’t have time or space to experiment, so they prefer to bring in subject matter experts or SMEs to justify what they want to do anyway. The SMEs take the blame when things go bad as they usually do.
Experts give their opinions based on gut-instinct or prior experience, which is usually out of date or based on assumptions that were never fully tested in the first place. Experts make a reputation for themselves by exuding confidence so they assert that they know their solution is the best one imaginable. The SMEs wouldn’t be sought out otherwise.
When asked to support their reasoning, experts can normally cite countless statistics and case studies that back them up, thanks to selective research and confirmation bias. Confirmation bias is the natural human tendency to pay attention to only the data that confirms what you already believed in the first place.
It requires a carefully structured experimental design to counter the effects of confirmation bias. That is one of the primary reasons that even great innovations fall apart on execution. A study by McKinsey found that although executives at global firms agreed on the critical value of innovation, 94 percent of them were not satisfied with their own results and they lacked clarity on how to improve the situation. Next time you are speaking with other executives about innovation, remember that they are probably thinking that you have a 94 percent chance of screwing it up.
What innovation teams that fail usually lack is a functional Universal Undo Button, in the form of hypothesis-based innovation testing. Built on their successes from rapid and incremental iterations, companies grow more innovative, more agile and pull further ahead of their rivals. They reduce risk and take advantage of increasingly narrow windows of customer demand.