A while ago, I discovered something that completely changed how I think about control groups. I was working for a streaming client, running their win-back campaigns. If you’re familiar with streaming, you know the pattern – users subscribe for a month or two, binge a show or two, then cancel. It’s a constant battle to bring them back, which is why we invested heavily in win-back campaigns through email and push notifications.

To measure success, we used what I call a three-layer control group approach:

1. A global control group (5-10% of users) that never received any campaigns
2. Campaign-level control groups, randomly selected for each campaign
3. The treatment group – everyone else who received our campaigns

There were two reporting cycles  –  weekly, where we’d focus on individual campaigns, and quartely, where it was focused on compounded, high-level impact.

Weekly results looked great – some showing up to 150% increase in subscription renewals. Then came the quarterly executive review…

The Problem

At this point, we’d step away from campaign-level control groups and use the global control group to analyse the long-term, compounded impact of all campaigns we did in the last quarter. 

We were ready to present our big win: “We brought back 50,000 additional users, generating $2M in extra revenue!” Except… the numbers showed almost no positive impact.

First instinct? The numbers must be wrong – the process, the setup… Something had to be wrong.

But no. After redoing all reports in multiple ways with multiple data sources, everything matched – individual campaigns showed positive effects, while the cumulative report showed no positive effect.

Why It Happens

Imagine you have an orange. You cut it in half:

• One half sits on the table (your control group)
• The other half you squeeze repeatedly (your treatment group)

The first squeeze yields lots of juice, and comparing it to the untouched half shows great results. But what happens when you keep squeezing that same half? Each squeeze yields less juice, while the untouched half still holds its original potential.

This is exactly what was happening with our user groups:

• The global control group maintained its natural conversion potential
• Our treatment group, repeatedly “squeezed” with campaigns, gradually lost its conversion potential
• Campaign-level control groups, being part of the broader treatment group, were equally depleted

With constant “squeezing,” the treatment group’s structure and behavior changed, making it non-representative compared to the control.

The Problem Visualized

Let’s illustrate this visually:

Here’s what happens to conversion rates over time:

• Global Control Group: Conversion rates remain relatively consistent, driven organically with natural ups and downs due to seasonality or external effects.
  
  
• Campaign-Level Control Group: Starts consistent but trends downward. Over time, as the “juice” is squeezed from the treatment group, their conversion potential decreases.
  
  
• Treatment Group: Shows oscillating conversion rates — big peaks on campaign days and dips afterward. Over time, fewer users remain likely to convert, causing an overall downward trend.
  

Key moment: The average conversion rate of the treatment group eventually falls below the conversion rate of the global control. That’s when you’ll see positive results at the campaign level but negative results compared to the global control.

The Solution: Control Group Rotation

The solution is to reset your global control group after a predefined period. When starting an initiative, define the audience, select the control group split, and decide how long the initiative will last. After that period, conduct your analysis and randomly select a new control group to normalize the audience.

Time period you’re gonna select is specific to your case. It will usually align with business timelines and objectives. For example, if you have an offer that lasts two weeks – you’ll isolate a control group for two weeks. 

Pro tip: Don’t keep the same control group for over 3 months. Anything beyond that starts manifesting big differences in segment behavior and creates analytics problems.

The Conclusion

Rotating your control groups ensures your data stays reliable and actionable, preventing misleading results from campaign fatigue. By resetting at regular intervals, you’ll maintain balanced comparisons and uncover the true impact of your efforts. Don’t squeeze the same orange too long — fresh control groups are the key to better insights!