August 20, 2014

While experts and pundits have pointed at the many business opportunities in Big Data, there’s a growing concern about the value of Big Data projects. The answer might be Predictive Analytics.

Traditional analytics, business intelligence and reporting are mostly describing the past, using historical data gathered over time. By speeding up data collection, processing and analysis, you can analyze faster and report sooner to management – but still after the fact.

Forecasting - predicting future events with a degree of certainty - is a huge boost to most types of businesses and organizations. Weather forecasts are an obvious example: we already know what the weather was today, but we would definitely like to know how it’s going to be tomorrow.

Financial traders want reliable predictions of future stock prices, commodities or exchange rates. Businesses need accurate forecasts of demand for their products or services to plan production and staffing. Sales and volume forecasts for, say a shopping mall, could take into consideration multiple factors like social media mentions, public transport disruptions, local events, weather and season. The Internet of Things (IoT) will undoubtedly lead to an explosion in raw data collected from sensors and gadgets of all types, providing additional analytical dimensions.

Business intelligence and analytics haven’t quite made it there yet, although the stakes and potential gains in many cases are high. A major reason is that predictive analytics is very complex. What appears to be proven correlations between indicators may actually be just coincidence. Factors that are assumed to be important can turn out to carry little or no weight in end results. Inherent randomness may be overlooked, leading to erroneous conclusions.

Currently, a lot of work is done to improve predictive capabilities. One interesting track is machine learning, recognized by – among others – Microsoft, who recently announced their new cloud service, Azure ML. The promise of Azure ML is that you don’t have to be a data scientist to use it – Microsoft already did the heavy lifting for you. By applying Azure ML methods on your datasets you’ll be able to create predictive analytics services for your own purposes, hosted in the cloud.

Data scientists are working on resolving various complex issues in predictive analytics, as this article, as this article from data mining software reviewer Software Advice, on ways to test predictive models shows. These methods mainly apply to analysis of structured and quantitative data – “number-crunching” – but some are also applicable to unstructured data and text analytics. Clearly, human judgment is needed in understanding, developing and verifying predictive models. As of now, machines are not (yet) capable of learning or predicting much on their own.

Still, the value proposition of Big Data won’t be quite fulfilled until predictive analytics has become useful, usable and valuable. Getting to that next level requires not just massive amounts of data and computing power, but smart people with tried and tested methods.

TextOre, Inc.