This guide will provide you with the necessary information to understand the “Increase in” also known as Incrementality or uplift metric.
Watch the explainer video or continue to the questions below for more information.
Video Transcript
Welcome! If you’re watching this video, you’re probably wondering how does Optimove calculate the “Increase In” metric. You’ve come to the right place. In this video, we will explain all about it. 😊
Optimove allows you to understand the incremental impact of any campaign, stream or even a complete marketing plan, by using the “Increase in” AKA “Uplift” metric.
An 18K increase in total order amount, for example, means that you generated an additional $18K by sending out the campaign.
Optimove calculates the “Increase In” metric by measuring the difference between the total order amount generated by the campaign, and the order amount that would have been expected in the absence of any campaign, represented by the behavior of the control group.
Here’s how it works:
We ran a campaign using a target group of 2,761 customers.
These customers are split between a test group that received the campaign and a control group that did not receive the campaign.
The campaigns’ KPI and response rate are statistically credible.
Step one is to find out what’s the total revenue - how much all 2,761 customers spent.
In our test group, there are 415 customers who made an order. Their average order amount was $67.
If you multiply 415 by 67, you get $27,805 – that's how much revenue the test group generated.
Now, let’s find out how much the control group spent.
36 customers in our control group made an order without receiving our campaign. Their avg. order amount is $31.
If you multiply 36 by 31, you get $1,116 - that's how much revenue the control group generated.
Together, the test and control groups generated $28,921 net revenue.
Now that we have the campaign’s total net revenue, we can move on to our second step:
using the control group's behavior to model the net revenue that would have resulted from all 2,761 customers, had we not sent the campaign at all.
Our control group customers have a 13% response rate, meaning that 13% of those customers made an order.
Their average order amount is $31.
Multiplying 13% by 2,761 gives us 358.93. Let's round this up to 359.
It is safe to assume that 359 is the number of customers that would have made an order had we not sent the campaign at all.
Now, let’s multiply 359 by $31, which is our control group’s avg. order amount, to get $11,129.
That’s the total expected net revenue if this campaign was not sent out.
Now that we have: the actual total net revenue and the expected revenue in case we hadn’t sent this campaign, our last step is to calculate the uplift generated by this campaign.
We need to subtract the campaign’s expected revenue if no campaign was launched from the campaign’s total revenue.
Therefore,
$28,921 - $11,129= $17,792
The campaign generated an additional $17,792 in net revenue.
Notice that this calculation is only possible with a control group. That is why using control groups in every campaign is so important. That’s the only way show the true uplift generated by your work.
Thanks for watching.
What is an Increase in/Uplift?
Increase in/Uplift – What does it mean? How is it calculated?
Optimove allows you to know the incremental impact of any campaign, journey or of your complete marketing plan via the “Increase” or “Uplift” figure. In this article we will focus on the campaign level, but the explanations hold true for journeys and complete marketing plans as they are built from multiple campaigns.
In short, the “Increase” figure answers the question: how much additional revenue (or any other KPI) did a campaign actually generate during its measurement period?
This figure indicates the uplift also known as Incrementality, beyond what would likely have happened had the campaign not been run at all, which is calculated by comparing the response of the test and control groups for each campaign during the campaign duration period. (For campaigns run without a control group, the “Increase” metric will not appear). Note that customers in the control group represent the baseline behavior of customers in this target group, meaning their behavior without receiving the campaign in question.
Is campaign uplift the same as revenue?
No. Campaign uplift and revenue, or any other KPI for that matter, are not the same.
-
Revenue is the actual money received from a customer in a given time period.
-
Upliftlso known as Incrementality, is a statistical quantity measuring the additional money spent by the customer due to a specific campaign. In other words, it explains the monetary difference resulting from sending or not sending a given campaign to a specific target group.
Since these two metrics are not one and the same, they may be substantially different. For example, an uplift may exceed the overall revenue. If a campaign generated a revenue of $1,000, it may be that the overall uplift was $2,000.
Here’s an intuitive example:
In a specific campaign, the average net revenue from customers who received the campaign was $100 and the average revenue from customers in the control group was (- $100). The test group deposited $100 on average, while the control group withdrew $100 on average.
In this case, the overall absolute revenue is zero (assuming the size and response of customers in the test and control groups are equal).
However, the uplift in this case is $200, since being targeted with the campaign caused customers to spend $100 instead of withdrawing (-$100). In other words, the difference between targeting or not targeting customers in this target group is $200.
Measuring uplift
What is needed to measure campaign uplift?
For you to be able to know the exact increase to a given KPI, such as revenue, of any campaign you must use a representative control group (read more on the importance of representativeness here). This is due to the fact that “Increase in” is the difference between the performance of a test group against a control group.
It is recommended to use exclusions across your marketing plan to limit the campaign exposure of customers to a single campaign. By using exclusions, you guarantee a cleaner experiment providing better accuracy and reliability of the uplift results.
For example, when using exclusions and having two campaigns: Campaign A and Campaign B, a customer can only be in either A or B. However, when not using exclusions you actually have 3 customer groups:
- Customers that received only A
- Customers that received only B
- Customers that received A & B
Although the above is a simple example, as the number of overlapping campaigns not using exclusions increases, so does the amount of possible permutations.
As permutations increase, it is harder to guarantee representative samplings since the target groups will be smaller for each permutation (i.e., there are less customers receiving A&B than receiving only A or B). In other words, as the number of campaigns a customer receives during the measurement period increases, it becomes more difficult to ensure that each “test customer” in each combination has an equivalent “control group customer.” This in turn impacts the validity of the measurement, since for high validity it is necessary that for each “test customer” there is an equivalent “control group customer.”
How is uplift calculated in a Test/Control campaign?
When conducting campaign analysis, Optimove considers two distinct performance factors:
- Response Rate: the number of customers that contributed to the uplift metric out of the total number of customers in the group (e.g., those customers who made a transaction at any time during the measurement period)
- Average Value: the average contribution to the uplift metric by customers in the group (e.g., average order value of all customers in the group who placed an order during the measurement period)
Many campaigns drive independent, or even opposing, effects on these two factors. For example, a deep discount on a specific offering is likely to increase the number of conversions, while simultaneously reducing the average revenue per transaction. In terms of the factors we’re discussing, such a campaign improves Response Rate among the test group’s members (compared with members of the control group), but decreases the Average Value.
Optimove calculates the campaign’s uplift in terms of the combined effect of both factors. This is how it works:
For each of the two factors, Optimove uses the Bayesian Monte Carlo method to calculate the probability that the performance of the test group is better than that of the control group. Probabilities above 90% are considered as an indication that the observed difference in performance between the test and the control groups is indeed due to the campaign, so such a difference should be considered a credible uplift calculation. On the other hand, less conclusive probabilities suggest that the difference in performance might not necessarily be attributed to the campaign – rather, there is a non-negligible chance that the observed difference is “random.” In other words, the difference might have been observed when comparing any two arbitrary groups of customers – so that this difference in performance should not be considered a credible uplift calculation.
A credible positive uplift means that the campaign was effective, successfully influencing customers towards desirable behaviors. Uplifts that are not credible – whether positive or negative – simply mean that there is not enough evidence to make a conclusion either way.
A table like the following, which appears on the Campaign Analysis page, presents the campaign performance metrics used to calculate the “Increase in” figure for the Revenue KPI of a campaign with a test group of 1,166 and a control group of 235. A total target group size of 1,401 customers.
What is an Increase in/Uplift?
Why do I need results to be statistically credible?
If a result is not statistically credible, then that result may be due to nothing other than chance – it tells you little or nothing and can be misleading. Including such results in your decision-making may lead you to make decisions based on random chance, rather than on the actual performance of campaigns, a situation that would likely lead to sub-optimal performance of your overall marketing plan.
Is the uplift’s statistical credibility impacted only by target group size?
No. There are three major factors in play:
1. The target group (sample) size: A larger number of customers leads to higher precision in the distributions of the Response Rate and Average Value for each group. This, in turn, pushes the probability that one group outperforms the other away from 50% (equal chances of either one being better than the other), and towards the credibility threshold of 90%.
2. The standard deviation of observed values (for more information on what standard deviation is, read below): The standard deviation is one way of measuring the level of “noise” present in the data (technically it’s a measure of the scattering of the data around the average, so a group of customers all having the exact same uplift value will have a standard deviation of zero). When the data is “noisy” (i.e., high standard deviation), only a very large difference between the test and control groups would be considered credible (an analogy: in a very noisy environment, only a loud shout will be heard, while a whisper will remain undetected).
3. The magnitude of the difference: With more dramatic differences in performance between the test and control groups (in Response Rate, Average Value, or both), the probabilities get pushed toward the credibility thresholds, allowing statistically credible conclusions to be reached, even with fewer data points and/or higher standard deviations. There is no particular threshold for each factor above that ensures a campaign becomes credible. Rather, it is the interaction of the three that comprise the probability of the test group actually outperforming the control group (or vice versa). Although occurrences and targeted customers do correlate strongly, since with more occurrences there probably will be more targeted customers, the number of targeted and control customers might still not be enough to reach credible results.
As such, aggregating more and more occurrences is not 100% guaranteed to increase the likelihood of the results becoming credible. It is important to keep in mind that campaign results might not be statistically credible because the campaign itself is simply not effective! If a campaign is, in fact, not successfully influencing customers to behave one way or another, increasing the number of recipients will obviously not lead to statistically credible results. It is also worth mentioning that the flip side of this phenomenon occurs in campaigns with huge sample sizes, say over a million customers. Such campaigns tend to be statistically credible, even with fairly minor test-control differences. In such situations, with such large sample sizes, the results may not be subjectively meaningful – such campaigns are prime candidates for further personalization/granularization.
What is the impact of target group size on campaign uplift?
The smaller the target group you use, the more campaign occurrences will be needed to obtain statistically credible results. There is no specific threshold for the number of recipients needed to achieve statistically credible results, as statistical credibility also depends on the mean and standard deviation of the customers’ behavior.
For example, take a campaign that is not successful, such that it embodies a “real” test-control response rate difference of, at most, 0.1%. In this case, you will probably need many more than 50 customers to get statistically credible results, as the group size needs to compensate for the campaign’s relatively weak impact. However, if the campaign works extremely well and embodies a “real” test-control difference of a is probably enough to achieve statistical credibility when analyzing the campaign results.
What is “Standard Deviation” and why is it important?
Standard deviation is a metric calculated from the data (just like the mean or the maximum are), that measures the tendency of the observed data points to not be neatly clustered together, but rather spread out from each other with no common anchor (data outliers, for example, increase the standard deviation).
Standard deviation can, therefore, be thought of as a “noisiness” gauge for data, and as a result, helps decide between what can be considered pure chance and what is more likely to be statistically credible. Here is an example: Let’s say a campaign’s test group spent an average of $120 during the campaign’s measurement period, and the control group spent an average of $100, yielding a $20 difference. If the calculated standard deviation was $100, then the result is still deep within the realm of the data’s natural randomness (because $20 < $100) so the campaign is far from having a statistically credible impact on the targeted customers. If, however, the standard deviation would have been $5, then the $20 would have far exceeded the level of natural randomness, making the test much more likely to be statistically credible.
What are the little green or grey curves in the Campaign Analysis table?
The Green or Gray icons are called “Gaussian Bells.” The calculation described above assumes that the differences in spending behavior between members of the test and control groups are statistically credible. If so, the Gaussian bell icons for both “% of Customers who made an Order” and “Average Net Revenue” are green, as seen in the example above.
However, when one of these differences is not statistically credible, the Gaussian bell for that metric is shown in grey. In this case, Optimove calculates the “Increase in” slightly differently (weighted average of both groups). If neither figure is statistically credible, then the “Increase in” figure will be shown in grey meaning that no conclusive “Increase in” result is available.
Interpreting Uplift and Attribution
How to interpret campaign uplift?
Uplift should be interpreted as the additional revenue (or any KPI) obtained as the result of sending the specific campaign to a customer when compared to a similar customer who was not targeted with that campaign. As a reminder, uplift is not equal to the numerical/absolute value of revenue or any other KPI.
The “Increase In” metric might appear in Green or Gray and with or without parenthesis.
- Green indicates that the test group outperformed the control group in the analyzed KPI and the difference in performance was statistically credible.
- Gray indicates that the result was not statistically credible, regardless of whether the test group outperformed the control group or not. This means the campaign had no attributable uplift to the measured KPI.
How does campaign uplift relate to attribution measurement?
Uplift is the unique marginal contribution of each specific campaign, making it the only solution to scientifically measuring campaign attribution.
For example, let’s assume you have two campaigns targeting a customer simultaneously. A birthday campaign and a newsletter campaign. At the end of the measurement period, you see that Jane Doe has spent $100 overall.
- The Birthday campaign shows an average uplift per customer of $5
- The Newsletter campaign shows an average uplift per customer of $10
The question is – how much money of the overall $100 can be attributed to each campaign?
Here’s the answer: $5 can be attributed to the Birthday campaign, $10 can be attributed to the Newsletter campaign, and the rest can be attributed to spontaneous behavior irrespective of any campaigns. Why? The $5 uplift from the Birthday campaign means that having been targeted with the Birthday campaign made Jane spend $5 more than what she would have spent otherwise, had she not been targeted with the Birthday campaign. This is regardless of any other campaigns she was exposed to (e.g., the Newsletter campaign).
The fact is that there is an equivalent and representative control group for each campaign. As explained by the diagram below, for each customer who received the Birthday campaign and also the Newsletter campaign (labeled “Both”), there is an equivalent customer who did not receive the Birthday campaign but did receive the Newsletter campaign (labeled “Newsletter Only”). This helps balance the effect of the Newsletter campaign on both groups and allows for a clean isolation of the unique and marginal impact of the Birthday campaign. The same is true for the uplift calculated for the Newsletter campaign.
So overall, on average, for a customer such as Jane Doe, $5 can be attributed to the Birthday campaign, $10 can be attributed to the Newsletter campaign, and the remaining $85 she spent (of a total of $100) can be attributed to spontaneous spending behavior that can be expected by customers who were not targeted with campaign A or B.
What is the ideal percentage of uplift to attribute to each campaign?
When a customer is targeted by more than a single campaign during a measurement period, there is no need to calculate a percentage of uplift to attribute to each campaign. As explained above, this is because the average uplift per customer of each campaign provides the exact attributable value to each campaign.
Does targeting customers more than once a day impact uplift? Does it incur a double counting of uplift?
No. As explained above, the uplift is calculated by comparing the test and control groups of each campaign individually. Meaning that Optimove is able to isolate the unique uplift generated from each campaign from the others. Put simply, customers’ spend is not counted multiple times in the campaign analysis and uplift calculations. However, it is recommended to exclude customers from concurrent campaigns to guarantee a cleaner experiment providing better accuracy and reliability of the uplift results.
Is there a difference between Uplift measurement for a single campaign occurrence and a series analysis?
No. However, by looking at a series analysis instead of a single campaign occurrence you are analyzing a larger customer sample (more customers have received your campaign than in a single occurrence) which provides the stronger statistical power to the test. Additionally, you are receiving a longer-term view of the impact of your campaign, which is better when making strategic decisions. When target groups are small, the test and control groups are consequently small. As a result, achieving statistical credibility of the uplift generated by the campaign in a single occurrence is difficult. Series analysis overcomes this issue by virtue of summing up the individual occurrences into a larger customer sample, allowing you to retain personalization achieved from small groups without losing statistical power and measurement accuracy.
If I sum up the uplift from individual occurrences, will the total uplift be equal to the uplift I would get by performing a series analysis on the same period?
No. Just as averages are not additive (an average revenue of $4 on Sunday’s occurrence and an average revenue of $6 on Monday does not mean the average of Sunday + Monday will be $10), uplift is also not additive. There is an element of aggregation when conducting a series analysis (namely the aggregation of customers from the different occurrences during the series’ time period), but that’s not the only component in the mix.
When conducting a series analysis, we group together the customers who were part of the test groups for the relevant time period, thus creating a new, “average,” test customer with a new average value. The same is done with control customers, thus creating a new, “average”, control customer with some new average control value. This in turn changes the average uplift per customer for the series’ time period.
Often an individual occurrence may not be statistically credible, for example, due to a small sample size, however, when conducting a series analysis the overall result may be statistically credible. This as well may impact the uplift, as the specific uplift formula used depends on whether or not the difference in the measured KPI, the difference in response rate, or both, are statistically credible.
Reports
Where can I see the “Increase In” metric?
As mentioned above the metric can be viewed on the campaign analysis page. Additionally, it can be viewed on the Campaign Analysis Page, in Mission Control, and in the Marketing Plan Explorer.
Campaign Analysis Page
At the top of the Campaign Analysis page, there is a summary “Increase in” figure which you can toggle for any KPI. This increase in the figure displays the value for a single occurrence or a series analysis depending on your selection. For more on Campaign Analysis click here.
Mission Control
In the Mission Control dashboard, there is a summary “Increase in” figure that aggregates the “Increase in” figures for all the campaigns that match your chosen KPI. Only statistically credible uplift results are included in this summary metric.
You can also add the Increase and Increase Per Customer columns to measure the impact of each campaign series and occurrence.
Why can’t the Mission Control performance summary give a figure which takes into account all KPI increases and decreases, and not just the statistically credible ones?
If a result is not statistically credible, then that result may be due to nothing other than chance – it tells you little or nothing and can be misleading. Including such results may lead you to make decisions based on random chance, rather than on the actual performance of their campaigns, a situation that would likely lead to sub-optimal performance of your overall marketing plan.
Marketing Plan Explorer
The “Increase in” metric presented in the Marketing Plan Explorer is the sum of the uplift results for all the campaigns run during the currently displayed period (week or month).
This total uplift relies mainly on the performance of the campaign as a whole during the specified period, instead of the sum of its daily instances (each square in the marketing plan. This calculation basically adds a stage in which for each campaign the entire week’s/month’s data of that campaign is put in one basket (all test customers in one basket, and then all control customers in one basket) and the overall uplift during that period is calculated. Only if this delivers no significant results will the system revert back to the same calculation done in the marketing plan, i.e., a simple sum of the daily (statistically credible) uplifts presented in the matrix in that period.
If, and only if, none of the individual campaigns nor any of the recurring campaign series composite uplift results were statistically credible, then the non-statistically credible composite results of all campaigns are added to the total “Increase in” displayed (in this unusual instance, the “Increase in” metric will be displayed in grey, indicating that it should not be relied upon for strategic decision making).
