This report empowers you with insights into the accuracy and effectiveness of Optimove’s Predictive Models. By providing a detailed comparison between predicted outcomes and actual results over time, the report enables better decision-making with insights into your customer base.
Navigating the Report
Choosing Your Focus
- Use the Predictive Model dropdown in the top-left corner to select which model to analyze (e.g., Conversion).
- Choose a specific snapshot in time to evaluate historical performance.
The table provides a side-by-side comparison of predicted versus actual outcomes, broken down by prediction categories (buckets). Here’s what each column represents:
Let’s use Risk of Churn as an example:
| Column Name | Meaning |
| Bucket | This represents the prediction category. Each bucket groups customers with similar predicted scores into a range of percentiles. |
| Customers in Bucket | The total number of customers in the bucket on the selected snapshot date. |
| Predicted by Model | The predicted number of customers expected to churn in this bucket, based on the model's probability scores. |
| Predicted (%) | The average predicted probability of churn for this bucket. |
| Actual | The actual number of customers in the bucket that churned. |
| Actual (%) | The percentage of customers in the bucket who actually churned. |
| Difference | The gap between the predicted and actual percentages. This is a direct measure of the model's calibration, or accuracy. The formula is: Difference = Predicted(%) - Actual(%). |
How to Interpret the Model's Predictions
It is important to understand that the model does not make a binary ("yes/no") prediction for each customer. Instead, it calculates a continuous probability score (a value between 0 and 1) that represents the likelihood of that customer performing an action (e.g., churning).
The 'Buckets' you see in the report (e.g., Critical, High, Medium) are simply quantiles used to group customers with similar probability scores.
Because the output is probabilistic, there is no strict notion of "false positives" or "false negatives." Instead, model accuracy is evaluated through calibration—comparing the predicted churn rate against the actual churn rate for each bucket. The 'Difference' field represents the calibration gap. A small difference indicates the model's predictions are well-calibrated and highly accurate for that group of customers.
Color Coding for Differences
To make interpretation easier, the Difference column is color-coded:
- Green: -0.05 < difference < +0.05 (Accurate predictions)
- Yellow: -0.2 < difference < -0.05 OR +0.05 < difference < +0.2 (Moderate variance)
- Red: difference > +0.2 OR difference < -0.2 (Significant variance)
Understanding Attribute Impact
Attribute Impact highlights which customer attributes (features) had the greatest influence on the model’s predictions. This section explains what drives the model's decisions, building trust in the results and pointing you toward the attributes that are most relevant for your strategies.
How Impact % is Calculated
Optimove’s predictive models use CatBoost, a sophisticated gradient boosting algorithm. To determine an attribute's importance, it measures how much that feature contributes to reducing uncertainty in the model's predictions. This is calculated using a method called log-loss reduction (also referred to as information gain).
Crucially, this importance calculation is performed for each model (e.g., Conversion) and for each distinct customer lifecycle stage within that model. The Impact % you see in the report is an aggregated value, summarizing the feature's overall importance across these different lifecycle stages. A higher percentage, therefore, indicates that the attribute consistently improved the model’s confidence in its predictions across your entire customer base.
Interpreting the Metric: Significance vs. Magnitude
The Impact % reflects an attribute's significance, not its magnitude.
- Significance (What this metric shows): This is the consistency of the attribute's contribution. A high Impact % means a feature is consistently important for making accurate predictions.
- Magnitude (What this metric does NOT show): This is the size or direction of the effect. For example, the metric tells you that "Days Since Last Activity" is very important, but not whether a higher value increases or decreases churn risk.
Example: If "Days Since Last Activity" shows an Impact of 12%, it means this feature was consistently one of the most powerful factors the model used to distinguish between customers who would churn and those who wouldn't. Its impact may fluctuate seasonally; for instance, at the start of a football season, this attribute might become even more significant for a sports-betting operator.
How the Model Learns
You can think of the model as playing a game of "20 Questions" to guess if a customer will churn. At each step, it needs to ask the best possible question (i.e., check a customer attribute).
"Information gain" measures how good a question is at splitting a mixed group of customers into purer groups. The model always picks the attribute that provides the highest information gain. Attributes that are consistently chosen for their ability to reduce uncertainty earn a higher final Impact %.
Understanding the Difference Over Time
While the table shows the prediction difference for a specific snapshot, the accompanying graph illustrates how this difference evolves over time. This enables you to track improvements or declines in predictive accuracy and spot trends that may indicate shifts in customer behavior.