Timely analysis of data provides significant competitive advantage and this lead to Hal Varian, the chief economist of Google to comment, “I keep saying the sexy job in the next ten years will be statisticians.” Facebook calls their data analysts ‘data scientists’. While on the surface retail is different from Google or Facebook, but on the data level, they have something in common: retail is rich in data, and teasing insights out of large volumes of data requires a scientific approach. And one hallmark of a scientific approach is taking a hard look at facts, and making rational decisions.

The Unreasonable Effectiveness of Data

In a recent paper, a trio of Google researchers distilled their findings from trying to solve machine learning’s most difficult problems: “simple models based on lots of data trump more elaborate models based on less data”.

The most elaborate model possible is the human mind, working on ‘gut feel’. It can take into account a number of different variables like weather, morale, stock-market shifts, standing of the local baseball team. But in terms of data, an individual is usually restricted to the 50,000 rows in an excel spreadsheet. Also, the model is highly emotional and unpredictable.

Fitting Models to Data

The scientific approach is to create a simple model (the variables are usually restricted to seasonality, holidays, traffic, inventory and price). But the simple model is verified on a large volume of data, and the output is rational and predictable.

The diagram on the left shows how we can fit the actual sales (in black), with a model (in red), based on Seasonality, Holidays, and Price. While there can be many more variables that can be used in the model-fitting, it is important to remember that not all variables have equal predictive capabilities. A model that does a great job in fitting the data, by cannot be used to project into the future has very little practical use.

The other important factor to keep in mind is that correlation does not necessarily mean causation. Real life data is complicated, and often incomplete. It takes careful analysis before true causal factors can be teased out.

The science of retail is evolving, and Forecast Horizon delivers actionable scientific insights, through the use of complex algorithms running on the cloud computing platform.