Shedding a Light on Dark Data

Published 14 August 2020 by Philippa Gamse

The digital world generates an enormous amount of data – in fact, over 2.5 quintillion bytes every day, with the 4.57 billion active internet users each creating 1.7MB of data every second!  Some of that huge sea of information is coming to your business, which represents both a major opportunity for learning and improvement, and a major challenge in defining and implementing best practices to analyse and interpret it all.


The challenge is real – estimates of actual amounts vary between somewhere over 50% and somewhere over 70%, but it’s generally agreed that the majority of this data goes unused by most businesses – hence the term “dark data”.  There are two types of “dark data” – information that businesses know that they have but which they fail to take advantage of, and data that they’d like to have, but which they can’t be sure has been captured.


One of the key reasons given for this problem is that companies lack the tools and processes to accurately acquire, correlate and analyse the data that they need.  Information may be kept in many different systems across the organisation, in different formats, with duplications and inconsistencies.  Data often turns out to be incomplete when a company tries to apply it because there are gaps or missing elements in what has been collected.

As a result of all this, companies might overspend on marketing or advertising campaigns without realising the low return on investment, or they could miss significant opportunities for more personalised communications with customers and prospects that generate increased sales and revenue.  Patterns and trends in product sales which could inform supply chain and logistics planning can be overlooked.  Customer service teams might lack valuable insights about common issues that are causing reduced customer satisfaction and retention levels.

And to make it all worse, there can be excess focus on “vanity metrics” such as raw numbers of website visitors or social media followers, which don’t say much about actual benefits – lots of people “liking” posts are of little use to a business unless they buy something or otherwise fulfill that company’s goals.


If this sounds quite overwhelming – it is!  Or rather, it can be without a well-thought out digital measurement plan.  A good plan should be developed across the whole organisation, allowing you to align business objectives throughout all departments, and creating a common framework for defining goals and focusing on relevant and appropriate performance metrics.  You can agree on the most important metrics to focus on at one time so that you can maximise opportunities for testing and optimisation without getting distracted.

A centralised data hub can be designed with provision for ensuring completeness, accuracy and security of information, while removing duplication and allowing all teams to see a standardised, holistic view of your customers.  At the same time, you can identify the data points that you’ll need, together with when, where and how they should be collected.  Baseline measurements can be taken so that everyone knows where you’re starting from and how much progress you’re making.  


Einstein famously said: “Not everything that can be counted should be counted”.  It’s easy to feel overwhelmed by data, or to get bogged down in the weeds of looking at the wrong things. 


Our new lesson on Digital Measurement will show you the benefits of creating a comprehensive and company-wide digital measurement plan, and will walk you through what’s involved in effective implementation.  

We’re not saying that it’s easy – but we can certainly help you shed some light on all that “dark data”!

Share this article