Hi,

My conversations with your peers are always interesting. Recently, some have focused on how to best set up experiments to measure CRM strategies, specifically in scenarios where the same customer can (potentially) be part of the same target group multiple times.

Why is this noteworthy? Well, for starters, if you have campaigns with non-recurring customers, then the answer is easy. But in a world where customers can be part of the same target group more than once, measuring the impact of campaigns associated with that target group can be quite a pickle.

So here we are, and hopefully, this Optimove Magazine helps you easily answer this question moving forward. But first some ground rules.

No matter your customers’ “enrollment” in your campaigns, you should always keep in mind that your experiment design will change based on whether the campaign should drive immediate action or nurture a customer over time.

Examples of immediate action campaigns are abandoned cart reminders or time-sensitive incentives for a new deposit, while those for nurturing campaigns can be responsible gaming practices or a new collection drop announcement.

So let’s get to it, shall we?

Shai Frank, VP of Product

The good ol’ simple Test/Control

Let’s start with an easy scenario as a warmup. If you are planning on sending a one-off campaign that should drive immediate action from your customers, then simply create a campaign with a control group.

Optimove will automatically select a statistically credible representative group of customers and withhold them from the campaign. How representative? Well, Optimove divides your target group into various micro-segments and chooses customers from each one. The number of customers chosen from each micro-segment is proportionate to the size of the micro-segment compared to the size of the entire segment. The result is that every single customer in the test group has a corresponding customer in the control group.

This setup will allow you to scientifically measure the incremental impact of your campaign on any of your KPIs. For more on control groups, this Academy article has everything you need to know.

Stepping up into series analyses

What happens when you have a recurring campaign where each occurrence should drive immediate impact, but for which you also want to measure the incremental impact of all occurrences together?

For example, if you send a weekly promotional roundup and you’d want to know if it is driving more transactions. The campaign would take place every week, and you’d probably want to know how each one worked. But you should also be interested in how many transactions these campaigns generated over a specific time period, let’s say over a month, quarter, and more.

Set up a recurring campaign with a control group and use Optimove’s Series Analysis to view the total impact. You can even exclude outliers to improve reporting strength.

Optimove will potentially change the customers in the control group with each new occurrence. Why do you ask? Well to maintain the highest level of representativeness of course. Customers who receive this campaign are by default different than those who don’t (the difference being they’ve been exposed to this campaign). Without changing the control group, over time it will no longer be representative of the customers receiving the campaign. Thus, recalculating the control group each time the campaign runs, ensures that each customer receiving the campaign has a similar customer in the control group. This is the opposite of anchoring the control group when you first created the campaign and using that same one for every single occurrence, no matter what changes your customers go through.

One control to rule them all in Streams

While the first two setups took on immediate impact campaigns, what about compounded impact ones? For example, those where a series of messages are sent to drive customer behavior, with such behavior wanting to be attributed to the whole series?

In these cases, a consistent control group might be the best option for you, and thus you should use Streams.

Streams create a consistent control group from the outset, ensuring that your customers are separated throughout. You can also group more than a single target group and campaign under a Stream allowing you to measure their compounded impact.

Of course, adding control groups to each campaign in the Stream is an option as well. This is if you have campaigns that should be measured for their immediate impact, but that together should be measured for a long-term goal. For example, when you want to test your big event strategy and also the impact of each touchpoint within it.

Your CRM experiment setup cheat sheet

To bring theory to practice, we’ve put together the below cheat sheet for you. Whether you are targeting the same group of customers or different target groups each time. Whether these are campaigns with an immediate impact or a compounded one, the below should help you navigate your CRM experiment setup.

> From our blog

Need Convincing on using Control Groups?

We know some might be “cautious” to use control groups. If that’s the case in your company, this blog is exactly what you need.