Economist Robert Solow famously said ‘You can see the computer age everywhere but in the productivity statistics’.
It’s felt for some time like a similar aphorism could be made of Bayesian analysis in data science. Lots of people have said they are or want to be doing it, but few actually are.
Now, with the advent of various tools and a rising tide of awareness, the ‘Bayesian age’ seems to be upon us.
But what is it and why should you care? In this post we describe what Bayesian analysis is and give three benefits for considering incorporating it into your business analytics.
I know what I know
Bayesian logic is all about combining information in the best way in order to learn things about the world.
Information can be in the form of data or so-called ‘prior beliefs’; that is, things you just know. For example, you might be extremely confident that increasing the price of your product decreases how much of it you sell. You don’t need data to tell you the relationship is negative.
Bayesian inference is all about combining and weighting these pieces of information in an optimal way.
That’s all very nice, but why is this useful?
There is no doubt that going Bayesian is challenging. But the benefits are significant. Here are three of them:
1. Make better predictions with small or sparse data
The phrase ‘big data’ sounds a bit rusty now. For a good while it was making many organisations self-conscious. They’d ask us questions like ‘is our data ‘big’?’, ‘how can we start using big data?’ or ‘how much value will machine learning generate if we can get some big data?’.
The truth is that most valuable analysis is done on smaller data sets, with lots of gaps, outliers and errors.
Bayesian methods thrive in this context. They allow you to squeeze as much information as possible out of your data (and elsewhere) and to draw your models towards more realistic conclusions.
So-called ‘hierarchical’ or ‘multi-level’ Bayesian models in particular allow you to pinch information from where data are plentiful, for example well-established price elasticities on existing product lines, and use it where data are scarce, for example on a new product line.
2. Incorporate valuable non-data information
It can be hugely powerful to complement your model with other, non-data information. This could be in the form of a general feeling about the relationships between variables, or insights from previous analysis, whether your own or someone else’s.
Bayesian analysis considers your ‘prior beliefs’ and the data, and with how much confidence both are expressing their view, and combines them to provide insights.
3. Get much richer insights
And the results you get back are much more extensive. This can be almost overwhelming. Non-Bayesian analysis typically provides ‘point estimates’ - single numbers representing the average impact of X on Y implied by the data sample. Bayesian models provide distributions.
These distributions can then be used to tell you things like ‘what is the probability a 10% increase in price will lead to an increase in profit?’. Or ‘what minimum marketing spend is required to have a 95% chance of increasing subscriptions by 10,000.’
Bayesian methods can be extremely powerful and often solve shortcomings in more traditional, though perhaps easier approaches.
DS Analytics help organisations integrate Bayesian approaches into their analytics. We build models that help our clients learn from their data and provide training in Bayesian analysis for analytics teams using R and python.
Get in touch to find out more!