The key to determining the effectiveness of any customer marketing campaign is the proper use of a control group. A control group also known as a holdout group, is a subset of the customers you’re targeting with a particular campaign who will not receive the campaign.
Using control groups, Optimove can determine the actual monetary uplift of your marketing campaigns. By testing many campaigns, and keeping tracking the true effectiveness of each, you will be able to gradually optimize all your marketing campaigns for maximum response.
In this article you will find the answers to frequently asked questions about control groups:
Introduction to Control Groups
What are control groups?
A control group, or holdout group, is a subset of the total group of customers being exposed to a test. In marketing, control groups are used to measure the impact of a specific campaign or customer journey. Specifically, control groups are the customers you are targeting with a particular campaign who will not receive that campaign. The counterpart of control groups is test groups which are the customers you are targeting that will receive that specific campaign.
For example, if you are sending a particular offer to a segment consisting of 1,000 customers and using a control group size of 10%. Your campaign will only be sent to 900 members, the test group, while 100 customers will be set aside to be your control group. Those 100 customers will not receive the campaign at all.
How do control groups work?
Let’s say that you are sending a particular offer to a target group consisting of 1,000 customers. When you set up the campaign, select a control group size of 10% so that Optimove will send the campaign to 900 members of the target group (this is called the “test group”), setting aside a randomly-selected subgroup of 100 customers (called the “control group”) who will not receive the campaign. Once the campaign is over, Optimove determines the true effectiveness (monetary uplift) of the campaign by comparing the revenues generated by the test group with those generated by the control group during the campaign’s measurement period.
Why are control groups important in marketing?
Using a control group in marketing campaigns is the only way to measure the incremental impact, also called uplift, of a campaign or series of campaigns on a measured KPI. Without control groups, you would never know if your campaign was effective, ineffective, or even detrimental. By measuring the difference between the test and control groups, both in response rate and the chosen KPI, you can learn the true impact of each marketing activity.
For example, consistently using control groups in marketing campaigns reduces the possibility that you are cannibalizing sales that would have happened anyway. Cannibalization occurs when customers who did not receive a communication or promotion end up purchasing your products, implying that whatever was offered to the test group customers was unnecessary and costly for your business.
Why do some marketers not use control groups?
Many marketers believe that using control groups is equivalent to leaving money on the table. However, since not every campaign created works and not every promotion is necessary, without control groups marketers cannot differentiate good campaigns from bad ones. Other marketers believe that if a campaign has proven to be successful, they shouldn’t miss out on the potential revenue that would otherwise be generated through sending the campaign to the entire segment. Though understandable, this assumption fails to consider possible changes in circumstances that may affect customers’ behavioral patterns and preferences, as well as the need to constantly track campaign results to ensure that they develop positively.
Using Control Groups in Your Campaigns
When are control groups beneficial for marketing campaigns?
Control groups can not only help marketers learn the incremental impact of their campaigns, but also quickly identify changes in customer behaviors that require adjustments to current marketing campaigns and journeys. By identifying the changes in customer response and incremental impact on KPIs, marketers can adjust or completely stop their current efforts.
Below are a few common situations in which control groups can help marketers discover a need for such changes:
- Unforeseen Market Events or Changes in Customer Preferences – A new campaign by a competitor or a sudden change in customer behaviors can quickly reduce your customers’ response rate and the order value of your existing customer base. This may require a stronger incentive or an alternative type of offer that the competitor is not offering.
- Repeating Offers – Offers may not be as intriguing to customers who have seen them repeatedly. This may require a refresh of the campaign template or a need to sub-segment the entire audience into smaller segments to separate those that are still responding to the campaign from those who are not. Targeting the latter with alternative offers.
- Overlapping Offers – Overlapping campaigns can cause a decrease in response rate due to customer fatigue. In this case, the use of control groups may help first uncover the existence of overlapping campaigns and then determine which ones are unsuccessful.
- Irrelevant Offers – Targeting customers with irrelevant offers, can lead to them becoming indifferent to your campaigns, significantly reducing engagement. Irrelevant offers are mostly the result of human errors that should be prevented. However, when they do happen, control groups can surface the problem quickly as they don’t only focus on vanity metrics such as Open Rates but also report on the direct financial impact of campaigns.
When are control groups not necessary in marketing?
There are a few situations in which using a control group is recommended but not completely necessary. These are situations in which either marketers aren’t interested in seeing the ROI of a certain campaign or there is no risk of a change in circumstances that may lead to a change in customers’ responses to the campaign. Some examples include:
- Reminder campaigns – in which only the customers who were part of the initial test group receive a reminder of the campaign.
- Non-trackable campaigns – if you cannot track the responses of your customers to a marketing campaign, for example when targeted customers are used only as a reference point (e.g., Facebook Lookalike Audiences). In these cases, there is really no need for a control group since the responses to the campaign will come from customers outside your initial segment.
- Educational campaigns – some marketers prefer that all customers receive educational campaigns. While this is understandable in situations such as additional purchasing platforms “Sign up to our new App”, it is important to remember that even educational campaigns or newsletters can show uplifts between the test and control groups since they are bringing awareness to the brand, causing customers to make purchases.
How does Optimove select control groups?
It is very important that control groups are a representative sample of the overall population receiving each campaign. To ensure this, Optimove takes into consideration the control group size and its randomness.
The first step is to ensure the control group size is accurate. As a rule of thumb, campaigns targeting fewer customers require a larger percentage of the recipient population to generate a valid control group, whereas campaigns targeting more customers can set aside a smaller percentage. For 10,000 customers, a control group of 5% is sufficient, while for campaigns targeting less than 2,000 customers, it’s a good idea to use 10%-20% instead.
There is an additional factor to take into consideration when deciding upon the size of your control group: your expected response rate. When you expect particularly low response rates for a specific campaign, such as when sending an offer to long-dormant churned customers, you will need a larger control group in order to obtain statistically significant results. On the other hand, if you expect a particularly high response rate, such as when sending a special promotion to your best customers, a smaller control group will be sufficient.
The second factor is the control group’s randomness. For you to accurately report on the incremental impact of your campaigns or journeys, the control group must be a representation of your target segment. As a result, Optimove divides your segment into various micro-segments and chooses customers at random from each one. The number of customers chosen from each micro-segment is proportionate to the size of the micro-segment compared to the size of the entire segment.
As already mentioned, the idea is for the control group’s customers to be similar to the members of the entire group, and thus represent what the entire group of customers would likely have done had they not received the particular marketing campaign being tested. The approach of randomly selecting a proportional number of customers from each micro-segment is an ideal way to accomplish this.
Optimove can optimize the control group size, reducing it as campaign occurrences take place. For more information, click here.
Is there a recommended control group size?
We usually suggest setting a control group of 5%-10%.
That being said, there are several factors to take into account when setting a control group to obtain statistically credible results; the target group size, the expected response rate, and the control group’s randomness. To read more, refer to the question: "How does Optimove select control groups?".
It is important to note that setting a control group percentage that will lead to a number of customers that is less than 1 will result in no customers being included in your control group.
For example, assigning 5% to the control group from a target group of 7 customers will lead to 0.35 individuals being assigned to the control group (5% * 7). As a result, the campaign's control group will include 0 customers. To ensure your campaigns always include a control group, ensure that the control group allocation always results in a number higher than 1.
To make things simpler, Optimove can optimize the control group size for you using the Control Group Optimization capability. For more information, click here.
How can I change the default control group size in Optimove?
Each Optimove instance has a default control group size set up. Although Optimove recommends leaving the default at 10% of the target group if you desire to update this percentage you can ask your CSM to do so.
In Triggered Campaigns, the test and control group ratio is set at the target group level. Read more here.
Should campaigns with small target groups use control groups?
A commonly asked question is: Should I use control groups for campaigns with a small number of recipients, given that the sample size is not large enough to obtain statistically significant results? This is a good question with a good answer! The answer is Yes and here’s why:
Small campaigns are usually run as part of a series of recurrences of the same campaign. A typical example is a “birthday campaign” set up in Optimove to send a special birthday bonus every day to those customers who are celebrating their birthday on that day. Because each individual day’s campaign will contain only a few people, the uplift calculations are unlikely to be statistically significant. However, Optimove can analyze recurring campaigns by aggregating the results of numerous recurrences and then calculating the combined uplift of the series. This will usually provide statistically-significant results that will indicate the true effectiveness of that campaign.
To analyze the aggregated campaign performance of a series of recurring campaigns, click the “Recurring Series Analysis” switch at the top-right of the Campaign Analysis page:
Note that you can select a specific date range for the recurrences to analyze together using the “Start Date” and “End Date” controls to the left.
Learn about the impact of the target group size on the campaign's uplift, here.
If a campaign is continuously generating successful results, should I stop using a control group?
No. You may think that if a campaign has proven to be successful, you shouldn’t miss out on the potential revenue that would otherwise be generated by sending the campaign to the entire segment. Though understandable, this assumption fails to consider possible changes in circumstances that may affect customers’ behavioral patterns and preferences, as well as the need to constantly track campaign results to ensure that they develop positively.
Using a control group will ensure that a red flag is raised if there is a change in customers’ response to the campaigns, which in turn, will enable you to continuously enhance and adjust campaigns to ensure high impact.
Should I be concerned about customers not receiving my campaigns because they are in the control group?
When using control groups, you may “miss out” on some customers receiving a specific campaign. The tradeoff, however, is worth it. Using control groups is the only way to show your work’s value contribution to your KPIs.
By maintaining control groups for all of your campaigns, you will be able to both improve your campaigns over time and report on your improvement with scientific evidence of your success.
In order to maximize your targeted customers, you can use the Control Group Optimization which will automatically minimize the control group size as your campaign begins to see results.