I read a very interesting article on smartplanet.com entitled “Why big data means job growth for non- data professionals” and from my perspective just this title summed the data explosion up perfectly. The vast increases in data if used correctly (and I mean a big IF) will have far more value to non-data professionals and to their organisations than anything a data scientist could produce. This is because data represents an opportunity to develop and accelerate human expertise far more than any spurious predictive model.
The quest for data scientists is still built on the beliefs that complexity is mathematically tractable. The moment a belief in a predictive model occurs then trouble begins, it’s the banking crisis model all over again. You can’t predict complex systems, but when you believe you can, you think you’re in control, that the model can do it for you, and that’s when human beings become passive. It’s a form of electronic social loafing, when predictive models become part of your cognitive dynamic, expertise takes a back seat and conflicting data or hunches get explained away. Just take a look at the Enron case study, all the data was there but the problems kept being explained away, with plenty of models, graphs and tables to help.
Better use of data does not lie in the hands of data scientists, it lies in the hands of experts, in any industry, using it to adapt and test their mental models. Good decision makers are not passive; they are adaptive and use vast amounts of tacit skills and heuristics to navigate complexity. Good decision makers also do not predict, they anticipate- cognitively, prediction is what leads you to lock your car keys in your own car, anticipation makes sure you don’t. The problem with following the examples of good decision makers is that very often they know far more than they can say. Unlocking this expertise and transferring it is true high value information but you’ve got to know how to do it, otherwise you could be following the wrong cues.
Big data and some of the latest BI platforms represent huge opportunities for experts in any field to operate at more adaptive levels which allow them to identify and lever risks rather than be buried by them. These technologies also represent the chance to help unlock tacit knowledge and turn novices into experts faster, and also broaden the range of expertise. When a focus is finally placed on how people actually use data to make decisions as oppose to how can more people use data to make decisions then we’ll finally see some real developments.
All decisions start with a hunch, intuition. Hours of research have taught me that, and the rules of thumb people generate to manage complexity are just as effective now as they have always been; we just need to identify them, support them and transfer them. So, forget data scientists, in about 5 minutes you can make almost anyone a better decision maker using data via Google, so focus on a usable, simple BI platform and then use it to support expertise.
Final thought- The data explosion represents huge opportunities for non-data professionals but they need to be used correctly and responsibly. Remember, the data sets are now are so vast you can prove almost anything you want via correlative statistics, so positive cases are of dubious value. At that point, ask an expert. I’ll explain further next time….