When you first start A/B testing, the feeling is phenomenal – almost every test brings improvements, and those improvements are sizable. But over time, that feeling fades. More and more tests fail to achieve positive outcomes, and even those that do have smaller effects.

What’s happening here is that all A/B testing improvements come with diminishing returns, or in statistical terms – you’re likely hitting a local maximum.

What is a Local Maximum in A/B Testing

Hitting a local maximum in A/B testing refers to a situation where you have maximized the performance of your current concept. By “concept,” I mean the subject of your testing – such as an email template, a landing page, or app screens.

Each concept has certain attributes that collectively set a theoretical upper limit on how well it can perform – for example, the highest possible conversion rate. If your tests no longer show significant improvements for multiple consecutive weeks, it might mean you’ve reached your concept’s maximum potential and need to explore a new concept with a higher ceiling.

Image: Current vs. Potential concept maximum

Having a local maximum implies that there is also a global maximum—a concept so perfect that it outperforms everything else. It’s a good idea to have a rough estimate of the global maximum for the metric you’re optimizing. For example, if you’re trying to increase the click-through rate (CTR) of marketing emails, it’s extremely unlikely that you’ll ever exceed 5%. In this case, 5% would be your estimated global maximum.

In reality, you’ll never know the true maximum since no one knows where the ceiling is. But if results stop improving – or improve very slowly – it’s a good sign that you’re close to the limit.

Getting over a Local Maximum with Radical A/B Testing

A solution to getting over a local maximum of your concept is to have a phase of radical A/B testing where you introduce one or multiple completely new concepts. This means you completely change all of the major attributes of your concept – for example, completely redesign your email or landing page. 

The redesign should still be informed by taking learnings from previous A/B tests, doing user research and/or competitive research, but it should be radical enough so it’s clearly obvious there is a difference in the approach. 

Image: Theoretical maximums of different concepts

By testing new concepts against the old, maximized one, you can identify which concept has the most potential to refine and optimize. When interpreting the results of concept A/B testing, a satisfactory concept is one that performs equally or better than your old concept. I’d argue that even slightly worse results can be acceptable because the new concept will have room for optimization, whereas the old one does not.

Optimizing the New Concept with Iterative A/B Testing

Once you’ve identified a promising new concept, it’s time to implement it as your baseline and begin iterative A/B testing. Iterative testing is what most of us do by default—progressively testing different variables of the concept, such as colors, copy, and images.

I recommend starting with elements that are likely to have the highest impact—like headlines and images—then moving on to smaller elements over time.

But wait, wouldn’t this lead to diminishing returns again?

Correct! The entire process is a loop where you start with a concept → iteratively test until improvements plateau → radically test new concepts → and repeat the cycle.

Image: A/B testing cycle

The Conclusion

Your final concern might be the thought that new concepts are also limited – there are only so many somewhat unique landing pages / email templates / app screens, etc. you can test, right?

Not exactly – new concepts are not driven only by your creativity to make new designs, but by technology advancements as well. Just think about how websites and apps looked 5 or 10 years ago compared to today. New development methods, design elements, and AI advancements continually create opportunities for fresh concepts.

In conclusion, if your A/B testing improvements have stalled, don’t feel stuck or frustrated. It actually means you’ve done a great job and maximized the potential of your current concept. You’ve climbed one hill, but there are likely taller ones on the horizon. All you have to do is look ahead and see which one is worth the climb.

Related read: What is Pilot A/B testing and when to use it