Bayesian MMM myth busting

In the process of writing ‘Making Effectiveness Work’ for the IPA, I came across a lot of misleading commentary on the challenges (and benefits) of Bayesian approaches to MMM. Here are 5 of them…

Myth 1: Bayesian modelling is too complex for most clients

It is certainly true that the stats & maths bar is high for Bayesian models. Ditto the coding challenges. It’s also true that Bayesian modelling can be a faff, with a more elaborate and involved workflow than generic MMM.

But these are challenges for the analyst, not the end user!

If done well, Bayesian models give the client much more useful outputs for guiding decision making.

Myth 2: Bayesian MMM is ‘cheating’

I think some have it in mind that Bayesian regression is exactly like OLS regression except it’s you not the data who’s in charge. If you were really determined to you could make your priors so strong that the data had no influence on the result. But if you’re going to do that you may as well skip all this coding and data and just use your keyboard to type the numbers you want into the results deck…

Priors are one of several benefits (and one of several brain power overheads). They can be uninformative, meaning all information is drawn from the data sample alone; they can be weakly informative, for example taking positive values only, or implying an ROI between £0 and £20; or they can be strong, centered around some specific value that you have good reason to believe is most plausible.

Myth 3: Bayesian MMM is only useful when you have patchy data

This again I think comes from a vague idea that Bayesian modelling is just normal modelling plus these things called ‘priors’. Even with lots of good quality data and uninformative priors, Bayesian models are still hugely useful. Most obviously because the outputs are distributions, not point estimates. Among other things this supercharges the power of scenario planning and optimiser tools.

Myth 4: Bayesian MMM cannot and should not be combined with experiments

This claim was recently made strongly here and in this lively LinkedIn thread. A 2024 paper by Google describes how this can be done in practice.

Experiments can provide the most robust read on the incremental, causal effects of advertising. If that information can be convoyed into your model’s priors in some reasonable way it makes sense to do that. Translating the generally narrower scope of a conversion lift or geo test into the parameters of a model isn’t straightforward, but can be done.

Myth 5: Bayesian MMM is always the right solution

I’ve spent a lot of time watching mcmc samplers limp towards the finish line. Or come down in the morning to a screen full of messages warning me not to use my model as it hasn’t converged.

Bayesian models require careful thought, doing your best to adhere to a ‘principled workflow’, lots of outputs wrangling, and lots of checks and sensitivity testing.

They aren’t the right solution to every problem and aren’t always the best solution for measuring ad effectiveness. I’ve found for MMM, it’s generally well worth the extra effort.