Validation is a critical step in ensuring the accuracy and reliability of your marketing mix model (MMM). This process tests the model's ability to predict outcomes based on unseen data, providing confidence in its forecasts and insights.
Backtesting
Backtesting is a key component of the validation process. This method involves removing a portion of historical data and training the model up to a certain point in time, such as three months ago. The model is then asked to forecast outcomes for the next three months. While the actual data for these months is known, the model has not seen it, allowing for a comparison between the model’s predictions and the actual outcomes.
Additionally, if the model can consistently make accurate predictions over time, even with changes in marketing budget and other interventions, it provides strong evidence that the model has effectively captured the true causal signals.
Validation Insights
In the Validation tab, a summary of the insights from all tests conducted over the last quarter is available. The data is presented in intervals of the last 4 weeks, 6 weeks, 8 weeks, 10 weeks, and 12 weeks. These insights include:
Predicted Revenue or Predicted Conversions: The outcomes forecasted by the model.
Actual Revenue or Actual Conversions: The real-world results during the validation period.
Estimation Error (%): The percentage difference between the predicted and actual outcomes.
By examining these insights, you can evaluate the model's accuracy and make necessary adjustments to improve its predictive capabilities.