I’ve often stumbled upon businesses focused on getting as many campaigns as possible out the door, but doing no or just very basic campaign-level reporting. Such a lack of measurement is like shooting in the dark and, more often than not, results in totally ineffective CRM strategies. For me, measurement should have equal, if not higher, importance than campaign execution. 

Recognizing how crucial reporting is in the world of CRM, this article dives into a foundational three-layer approach to measuring CRM strategies, from their immediate to long-term impact.

Measuring Immediate Impact

Immediate impact refers to the instant effect that a campaign has – usually hours or a day after it has been sent. This period of measurement is called the attribution window and usually doesn’t surpass seven days. This is the simplest scenario, and it’s measured using campaign-level control groups.

Essentially, we just need to select the control group size for each campaign we’re sending, and the CRM platform will randomly select a sample. This means that every control group will mostly be unique, as illustrated by the image below.

Image: Random control group selection for measuring short-term CRM campaign impact

*There will be some overlaps between users in different control groups due to the random selection. A-J clustering is only used for visualization purposes.

Measuring Short-Term Compounded Impact

Let’s imagine a scenario where you have a promotional offer that lasts for two weeks. A common strategy here is to send multiple campaigns announcing the offer, reminding users, calling the offer ending, etc. At the end of the two weeks, we want to make a report on the impact of CRM offer campaigns.

We can report on each individual campaign using campaign-level control groups, but just aggregating individual campaign data for the two weeks won’t work. The reason for that is that each campaign has a randomly selected control group, so a control group of one campaign will receive other campaigns, and vice versa.

A solution to that is having a static control group – a subsegment of our target audience that won’t receive any offer campaigns (it will continue to receive other types of campaigns). Such a control group is usually called a holdout. We’ll name it “offer holdout” in our scenario for more context.

By selecting our holdout group at the start of the offer, we can clearly measure the compounded effect of all campaigns at the end of the offer by comparing the metrics of the holdout (users who received no offer campaigns) and the treatment (users who received all offer campaigns).

Image: A combination of holdout and control group selection for measuring short-term compounded CRM campaign impact.

Unlike campaign-level control groups, this mechanism is usually not supported by out-of-the-box CRM platforms. It’s usually implemented by manually creating a segment of users that you will exclude from all offer campaigns. The most important thing is to ensure a random selection of our holdout – this is usually done by segmentation through user buckets.

User bucket numbers are numbers between 0 and 999 (or 9999, depending on the CRM platform) randomly assigned to every new user. 0-999 represents 100% of your audience, while 0-99 buckets would represent 10% of your audience. This is an easy way to randomly split any audience for holdout or A/B testing purposes.

Measuring Long-Term Compounded Impact

Long-term impact refers to time periods like three months or a year. These periods are usually milestones for businesses, as larger reviews are conducted and strategy decisions are made. For CRM, this usually means moving away from campaign-level reporting and moving to holistic topics like impact on revenue, retention, churn, etc.

Similarly to the last solution, this must be solved with a holdout but with a few twists.

For this application, we want to use a “global holdout”. This is a subsegment of all of your users that will not receive a single campaign of any type.

It’s important to reiterate the differences between a global holdout and the offer holdout.

Image: Global Holdout and Offer Holdout differences

This is often done by using user buckets, but many CRM platforms also offer out-of-the-box functionalities to achieve this. Some that come to mind are MoEngage and Braze.

Image: Global control group settings in MoEngage

An incredibly useful tactic here is to utilize multiple global holdouts. One universal holdout and a variable number of campaign-type holdouts.

• The universal holdout is a subset of all users and will receive no campaigns.
  
• Campaign-type holdouts are used to measure the compounded impact of a specific campaign type. For example, all winback campaigns or all weekly newsletters. A winback holdout would be a subset of all users who don’t receive winback campaigns but do receive all others. The number of holdouts is up to you to decide based on the complexity of your strategy and reporting needs.

Image: Advanced control and holdout structure for measuring long-term compounded CRM campaign impact.

The last thing to mention is the importance of audience rotation when using holdouts. Constant messaging or lack of it for the same audience will change the underlying structure and behavior of a segment, so after some time your holdout will no longer accurately represent the audience you’re measuring against. This will lead to analytics anomalies, but more on that in a case study I’ll write in the future.

You need to decide on a cadence at which you will rotate your holdouts – for me, that is every quarter. Rotation simply means changing the selected buckets or resetting the global control settings in your CRM platform.

The Conclusion

Some may have noticed that adding these additional control groups may reduce your audience reach. That is true, but I think the value of being able to measure and make informed decisions far outweighs the cost of audience reach.

In conclusion, effective CRM measurement requires a well-through structured approach that considers immediate, short-term, and long-term impacts. I hope this post provided a valuable insight into such an approach that can be universally applied to any business.

Image: Three layers in the CRM measurement process