Here’s a quick test to determine if reading this post will be useful for you.

Let’s say we’ve tested two different campaigns for the same audience, and we need to pick one to continue with. Pick the better performer:

Campaign A:

Audience Group	Users	Conversion Rate	Conversion Rate Lift
Control (10%)	1,000	4.60%
Treatment (90%)	9,000	5.78%	26%

Campaign B:

Audience Group	Users	Conversion Rate	Conversion Rate Lift
Control (10%)	1,000	2.00%
Treatment (90%)	9,000	2.84%	42%

If you picked campaign B, it might be a good idea to continue reading.

Key Concepts

Before I dig into the problem, I’ll define a few key concepts for this topic. If you’re familiar with them, just skip to the next section. 

Control group = a part of the target audience (often 5-10%) that doesn’t receive the campaign. It is used to measure the incremental impact of the campaign. That impact is called conversion rate lift.

Lift = a percentage metric representing a relative increase or decrease in conversion rate for the treatment group (users who received the campaign) over the control group. This term is called “relative change” in mathematics.

For the sake of simplicity, I’ll consider that all examples in this article are statistically significant.

The Problem

To many like me, the conversion rate lift is a single metric that determines the success of the campaign. Simply put, if:

• Lift is positive = the campaign is successful
• Lift close to zero = the campaign has no effect
• Lift is negative = the campaign has a negative effect on conversions

The lift size is also very important – the higher the lift is, the more successful the campaign is.

This is where I, my colleagues and clients have fallen into a trap before. Conversion rate lift shouldn’t be the only metric we use to evaluate the campaign’s performance because it only represents a relative change in conversion rate, not actual business results that are most often represented in revenue and/or conversions.

Let’s go back to the initial example, but take conversions into account as well.

Campaign A:

Group	Users	Conv.	CR	CR Lift	Added Conv.
Control (10%)	1,000	46	4.60%
Treatment (90%)	9,000	520	5.78%	26%	106

Campaign B:

Group	Users	Conv.	CR	CR Lift	Added Conv.
Control (10%)	1,000	20	2.00%
Treatment (90%	9,000	256	2.84%	42%	76

You can see that even though campaign B has higher conversion lift, it resulted in less added conversions than campaign A, therefore is less successful from the business perspective.

The Learning

A learning here is to always consider the starting conversion rate (control) – the lower the baseline conversion rate is, the higher the lift has to be to achieve the same absolute effect for the business.

For example, campaign B would have to achieve a conversion lift of 59% to achieve the same amount of added conversions as campaign A with lift of 26%. All because the baseline conversion rate of control is much lower.

Group	Users	Conv.	CR	CR Lift	Added Conv.
Control (10%)	1,000	20	2.00%
Treatment (90%)	9,000	286	3.18%	59%	106

This effect can be represented with a graph which shows lift dependency on baseline (control) conversion rate. The lower the baseline CR, the higher the lift needed to reach the CR of 14% (or any other arbitrary CR as an example).

The Conclusion

To conclude, calculating and using conversion lift to evaluate campaign impact here to stay because it’s still an incredibly useful metric to measure incremental value of some effort, but it’s always best used in combination with absolute metrics like added conversions or revenue that represent the real world value for the business.

Other than applying this knowledge in analysis, I think it’s also very useful in campaign planning and prioritization. It’s always useful to analyze the audiences you’re targeting prior to sending campaigns, so if you know that some audience has a really low baseline conversion rate, you might want to deprioritize that campaign because the lift you’d have to achieve for decent business results is really high.