Evidence Based Data-driven Decision Making: Achieving the Goal

The ultimate goal of architecting and building a big data technology infrastructure and analytical ecosystem is to optimize decision-making at all levels of the organization: strategic, tactical and operational.

Leaders understand the need for smart data and advantages that result from evidence based data-driven decisions. Yet most organizations do not have the right technologies, smart data sources, processes and talent to analyze and act on the right high quality data in near real-time.

The ability of an organization to make the best informed decisions is crucial in an increasingly competitive global economy. The collection and analysis of relevant high quality data is now an important part of both strategic and operational decision-making.

While many organizations are now attempting to adopt new tech and analytical methods to obtain meaning from data, and create an evidence based, data-driven culture, a significant majority are failing.

Why?

The simple reason is not understanding high-level tech, talent and process requirements to achieve specific goals. Moreover, it is tempting and usually wrong to attempt to execute a simple general purpose solution to complex problems. Three (3) key issues to consider:

1. Technology. Organizations desire modern data tech that provides a flexible, multi-faceted analytical ecosystem - to leverage both internal and external data to obtain valuable, actionable insights that allows the organization to make better decisions. Many have been sold on the Hadoop framework as a general purpose solution. In reality, Hadoop (primarily acts as a staging area, ETL tool and online archive) is simply one framework among many that is good at certain things and bad at other things. Whether to select the Hadoop framework depends on specific goals.

Unfortunately, a misunderstanding of the Hadoop framework/ecosystem and lots of new data tech is causing market confusion. The "Big Data" and "Hadoop" hype is causing many organizations to roll-out Hadoop / MapReduce systems to dump data into without a big-picture information management strategic plan or understanding how all the pieces of a data analytics ecosystem fit together to allow sophisticated machine learning algorithms and optimize decision making capabilities. This has resulted in the creation of a new word: Hadump - meaning data dumped into Hadoop with no plan.

I suggest a better strategy is to consider specific organization goals or use-cases and:

a) select the right technology for each specific goal and use-case;

b) select the right diversity of smart data sources;

c) develop optimal anaytical and decision-making processes to achieve specific goals; and

d) train or recruit the right talent.

2. Talent. Most organizations do not have the right talent mix to achieve specific data collection, processing and analytical goals. Many organizations will need a training and change management program that is often painful and time-consuming to achieve the goal of evidence based data-driven decisions. Many organizations will need to hire (either internally or externally) professional data scientists. It is often challenging to create high performing teams of data scientists and data engineers working with decision-makers and there are many organizational political impediments.

3. Processes. Many organizations will need to develop different types of decision-making processes (strategic, tactical and operational). The cost of collecting and storing data - and data analytics technology - has been significantly reduced and will get cheaper and cheaper. Yet the cost of analyzing the data for valuable, actionable insights is very high. While machine learning and automation will reduce cost in future, the formula of cheap, abundant data and expensive data science and business analytics will likely remain for some time.

Additionally, it is crucial to spend time and resources to develop both an information management strategic plan and decision optimizing processes. Data science knowledge and business processes detailing the collection, storage, analysis and distribution of the right smart data is the magic sauce that orchestrates the data tech ingredients.

Understanding specific organizational goals along with the right tech, talent and processes is the key to achieving the goal of evidence based data-driven decision-making.