Getting from data to insight

Survey after survey tells us that organisations are still struggling to fully leverage big data to get to insight and data driven decision making, despite investing billions. One major commonly overlooked reason for this is the difficulty of bridging the gap between pure data analysis and actionable insight.

The gap between data analysis and insight

Getting to insights is challenging because it requires both the science of data analytics and the art of data storytelling. Data storytelling for insight goes beyond pure data analysis since it involves communication tailored to an audience, such as a decision maker. It also goes beyond pure analysis because it must incorporate factors which the data alone cannot reveal no matter how much analysis is applied, such as idiosyncratic knowledge, uncertainty and even intuition.


The challenge of leveraging big data

A recent report by CapGemini The Rise of Insight Driven Business highlighted the importance of using data and insights for action to drive business growth. Yet in an extensive survey they also found that only 27% of senior executives believed their big data initiatives were successful. The survey highlighted data silos, and found that 52% of executives believed the IT development process constrained their ability to quickly create insights.

Of course it is not data or technology alone that constrains insight, but the inability to find the right people who are able to leverage it. As KMPG found in their global 2015 CIO Survey, 36% of technology leaders suffer from a shortage of big data analytics skills, higher than any other function asked about, and also high on the list, 29% say they are short of business analysis skills.

Clearly getting from data to insights is not easy, and needs investment in technology and people. But can the lack of investment alone fully explain why organisations are struggling to create data driven businesses? What happens when you have the data and the systems in place, but the actionable insight still feels thin on ground?

Telling stories with data

As Google have reminded us, data analysis is not about graphics and visualisations, it is about telling a story. This means identifying the audience, and speaking to their aspirations and concerns. Too many data analysts and analytic platforms create more information from data without clear direction as to precisely why the recipient, the decision maker, should be interested, and what they should actually do with it.

There is no shortage of data: organisations are drowning in it. Unfocused and poorly communicated analysis just adds more fuel to fire, more information for the time poor executive to navigate as they try to pinpoint what actions to take. Bridging the gap has to be about tailoring the analysis the needs of business decision makers. It is also about making sure the complexities of analysis are made easy to grasp and compelling, and essential points break through for those who need to know. This is data storytelling.


Insight often requires a leap in the dark

For data storytelling to work you need a receptive audience and many rightly stress the importance the creating a data driven culture (for example this post by McKinsey and this one by PWC). Typically the move to data driven culture is understood as move away from decision based on gut instinct and opinion. However as anyone who has tried to use predictive modelling for business forecasting will know, often it is impossible to remove all uncertainty and to resolve the decision problem you have without making some very big assumptions, and relying on intuition.

In fact data analysis and intuition are not mutually exclusive means to making smart choices, rather organisations must be comfortable with exploring problem solving from all angles and understand the limitations as well as the opportunities inherent in their data. Blind adherence to the dictates of data analysis which is patchy, has a margin of error or which cannot be generalised, may be even more dangerous than ignoring data altogether. A post by MIT Technology Review makes this point very well, notably in respect of risky decision making for innovation.

Summing up: Seven steps to get from data to insight

No online article is complete today without a list to summarise the thinking, so here goes – the first four are widely advocated as essential, while the fifth, sixth and seventh are about bridging the data to insight gap –

  1. Integrate your data
  2. Accelerate big data technology
  3. Invest in smart people
  4. Communicate insight by creating data stories
  5. Tailor your analysis to your business goals
  6. Drive for a data driven culture
  7. Tolerate some intuition and uncertainty

So there it is, in just seven easy to understand steps. In practice it will take years and is ongoing process. Getting from data to insight is a major overhaul of how your organisation behaves, requiring commitment, imagination and effort at all levels.

Gabriel Hughes

Can we help you become more data driven?
Please email us at

Getting from insight to action

While getting from data to insight is challenging, the final hurdle for creating value is about translating insight into actions. Data insight can actually help boost sales, customer engagement and critical business KPIs, but only if it is actually translated into action.

Insight to action is about people

Most organisations are sitting on huge and growing amounts of data, and then rely on analytics technology tools applied by teams of analysts to extract value. This arrangement means the analysts finds themselves as an intermediary between the decision maker and the data. The hand off from analyst to decision maker is therefore at the heart of an organisation’s ability to drive impact through actionable insight.

Many organisations cannot take full advantage of their data because they simply do not have the people or the culture to get the process to work. While big data grows in abundance, people with the skills to get from this data to insight are in short supply. The global 2015 CIO Survey by KMPG found big data analysis and business analysis skills in most short supply affecting 36% and 29% of those interviewed.


The demand for social skills and strategic understanding

Analysts and data scientists attract a certain type of person, the same kinds of people who are attracted to computer science and mathematics, and who are in short supply across all industries.

At risk of stereotyping, it has to be admitted that all too often this particular analytical skill set is associated with weaker of social skills and an inability to ‘zoom out’ and see the bigger picture. Many of us who have worked with statistical analysts are familiar with the experience of trying to get the analysis out the technical details and into actual implications and inferences. This is the problem of the big ‘so what’ that plagues so much analysis where despite many hours of work the question is still, ‘yes but what does this mean, what we should do differently ?’.

Research confirms that the folks who are in the highest demand of all are those who combine both technical skills and social skills. A 2014 University of California study of the US labour market found an ability to combine cognitive skills like mathematics together with social skills has become a key and growing determinant of career success.

A strategic view is also important, to understand the context of analysis. Strategic consultants often succeed where analysts fail because they approach problems from the top down and not the bottom up, meaning they start with the big questions that the organisation needs to answer and bend the analysis to that. The risk is they go too far in the direction of over claiming on the basis of weak data.


The data never speaks for itself

Somebody has to make the conceptual jump from analysis to implications, and then the organisation has to be ready to take action.The successful analyst has to be able apply themselves to the questions that the organisation needs to answer, and make strong recommendations when the price of inaction appears greater. This means creating data stories, and communicating results, which requires much more than technical analytic abilities. For it to work, decision makers in the organisation need to do their part to be ready to listen to what analysis is telling them, and put preconceived notions and internal politics aside.

Easier said than done

A review of the current evolution of analytics capabilities by the Gartner consulting group defined prescriptive analytics addressing the question ‘what should we do’ as the highest stage that an organisation can attain (for example in this presentation by Gartner). They rightly identify analysis which drives action as having the highest business impact and requiring the most advanced skill sets, to get beyond descriptive and predictive analysis.

Getting from insight to action is easier said than done.The difference between success and failure is how well people can work together to align around what the data analytics means for action. Organisations need to hire the best talent and develop the right organisational culture.

A culture of data driven decision making means that a junior analyst should feel comfortable telling a senior executive that their product is not right for the market, or their ad spend is wasted, or even that there is no good data to know what the best course of action should be. There should be no fear of ‘shooting the messenger’. Any debate about the right action to take should be grounded in a robust predictive model of the likely consequences of action for KPIs.

Your organisational culture must accept that even imperfect analysis is preferable to flying blind. You will find unexpected insights and opportunities along the way, and will learn the that process of trying to make data driven decisions is valuable in itself as everyone gets closer to the analysis.


Gabriel Hughes

Can we help you become more data driven?
Please email us at