Skip to main content

How Optimove Calculates an A/B/n Test winner

ben_t
  • Updated
In Optimove, you can create single or multi-channel campaigns, use control groups to measure incremental uplift, or leverage self-optimizing algorithms to ensure that each customer receives the best campaign and treatment.
 
Specifically, you can create A/B/n test campaigns to test different actions under a single campaign and determine which ones work better for your customers.
 
Additionally, you can run a Stream experiment to test a series of campaigns against another series of campaigns or against a control group, and examine a complete marketing strategy rather than one campaign in isolation.

How does Optimove determine that one action or Stream is outperforming an alternative version?

While both tests have different purposes and uses, the method by which Optimove determines the winner is identical in both.

This article will walk you through the process by which Optimove calculates the winner for A/B/n campaigns and Streams. We will also present two example use cases for the calculation.
 

How Optimove Determines the Winner

In each campaign, Optimove is tracking two metrics:

  1. Response rate (the rate of customers who took the desired action)
  2. Average KPI (a KPI specified by you)
The way Optimove calculates the winner is based on the statistical credibility of the two metrics.
When it comes to the above two metrics and their credibility, there are four possible scenarios:
  1. Response rate and Avg. KPI are both statistically credible
  2. Response rate and Avg. KPI are not statistically credible
  3. Response rate is statistically credible, while Avg. KPI is not
  4. Response rate is not statistically credible, while Avg. KPI is statistically credible
Each scenario results in a different calculation, as seen in the table below:
In the scenario below, both metrics are statistically credible.
 
We will first multiply both metrics for each variant and compare the results.

The variant with the highest result will be the winner.

In the screenshot below, variant A is the winner.
 
The 81% difference is determined by subtracting the winner’s result from the loser’s result. In this case: (0.000385-0.000214)/0.000214 = 0.81, or 81%.
 

Example Use Case

Winner calculation in an A/B/n test

In this example, a recurring A/B/n campaign was created to enhance customer migration from New to Active status. In this campaign, sent on July 29th, 2025, the Response Rate is credible, while the Avg. KPI is not. Action A was declared the winner.

As mentioned above, whenever the Response rate is credible, the Average KPI (Stake Amount) is not. The winner will be determined based on the higher Response Rate. In this case, A has a 37.5% Response Rate, while B has a 12.5% Response Rate. Thus, A is the winner.

Screenshot_2023-05-07_142914.png

Winner calculation in a Stream

In the example below, a Stream experiment was created to improve Net Revenue for New customers. Stream A includes a set of new Welcome Journey campaigns, while Stream B serves as a control group that doesn’t receive any campaigns.

The experiment has not ended yet, but Optimove has already determined that Stream A is the winner.

In this case, the Response Rate is not credible, while the Avg. KPI (Net Revenue) is plausible, and therefore, the winner is determined based on the highest Avg. KPI. Since A has an Avg. Net Revenue of $727 and B has an Avg. Net Revenue of $272, A is the winner.

Screenshot_2023-05-07_142936.png

Related articles