Data-driven decisions are fast becoming the norm for many businesses. The explosion in the amount of data generated and the ways/devices to record have made analytics and hence data-based decision making the ‘gold standard in 21st-century management.

The reasoning is solid. The more a manager knows about the company and the business s/he operates in, the better they are placed to make better decisions. 91% of businesses are reported to be investing in data for better decision making.

While the benefits of data are undeniable, are they always right? Is there any room for human intuition, the so-called ‘gut’ based decisions? More importantly, what happens when there is not enough or completely zero data to base your decisions on?

Havard Business Review gives a good example of the hit book turned movie Harry Potter and the Philosopher Stone. Author J.K Rowlings was initially turned down by 12 publishers but then churned out 500 copies. It was too original, too innovative for previous data in the industry to be used to judge its quality.

While famous books like The Lean Startup attempt to convert the process of innovation and business building into a science, the truth is data is often insufficient and sometimes blatantly wrong in creative endeavours.

Testing

To prove this, professors Oguz A. Acar and Douglas West did an experiment where they collected data from 122 companies. These were drawn from highly creative industries such as publishing, advertising and software development where survival is based on the quality of your ideas. They were asked to write down how they came to ideas and projects which they deemed to have been successful.

The results were astounding. Despite increased investment in data, managers did not rely more on it for their decisions than they did their own instincts or simple heuristics like tallying.

Furthermore, they found that relying solely on data is insufficient when choosing an innovation project. It was found to be a slow and laborious process whose only advantage was to very slightly increase the decision-making accuracy.

Managers who relied on intuition were found to be faster in implementation with no serious trade-off in accuracy. The inclusion of analysis in decision making was found to have no serious benefit in terms of both speed and accuracy.

However, this does not mean that data is completely useless in decision making. On the contrary, they found that managers who had more experience in their domain were better placed to make better decisions. This is obvious as such people can draw information from years of experience and therefore have a lot of ‘innate info’. For a novice, however, as a lot of startups find themselves to be, they are better off relying on data at least to prevent them from making big mistakes.

For those with the requisite experience, research suggests that they are better off making their decisions with their ‘gut’ feeling.