Have you ever wondered if your email campaigns are actually generating profits?
Sure, your analytics make it look like your email campaigns are having a big impact on your business. Engagement metrics like opens, clicks, and conversions are exciting, right?!
But, “Are these orders being generated profitability?” is a question that not many email marketers are asking themselves.
When was the last time you measured what customers would have done if you hadn’t emailed them that 25% off coupon?
Chances are your ESP isn’t giving you the answer either.
But fear not, this post will introduce the concept of holdout testing and why it’s the email marketers’ most effective tool for measuring the true profitability of their work.
What is holdout testing?
Holdout testing is the practice of regularly gut checking your email program to make sure that the campaigns being sent are actually generating true lift. “Lift” is defined as the incremental increase in revenue that is generated (or not generated) by sending a marketing campaign.
The test involves “holding out” a campaign for a “control group”, also known as a group of customers who are excluded from a marketing campaign. You measure the purchase behavior of the hold out group after 90 days, ultimately comparing their value as customers to the value of the group that received marketing.
A hold out test is designed to answer the questions:
• “How much lift is being generated by sending this campaign?” and…
• “Would these customers purchase anyway even if we didn’t market to them?”
Why do we need to know about hold out testing?
Consistently running holdout tests reduces the possibility that you are cannibalizing sales that would have happened anyway. For an eCommerce company that frequently sends promotions and/or offers to their customer list, it enables you to determine if those discounts are undermining the profitability of your business. The scary potential outcome for marketers is that the group of customers who did not receive a promotion end up purchasing your products anyway. If you’re offering a discount that would mean you’re giving away margin when it’s not necessary.
Here’s an example where this occurred:
In this above example $2.00 is being thrown away with each customer that purchases and when you scale that out to 1000, 10,000 or 100,000 customers it really starts to eat away at your revenue and profits.
Coming to the conclusion that email marketing programs are generating less profit than if no marketing had been done at all will not make you popular in the lunchroom, but will save your company’s hard earned marketing dollars (and may get you promoted).
How do you run a hold out test?
At Rejoiner, we build control groups using a randomly selected set of customers that represents 10% of the sample size we are sending to.
Here’s an example:
Let’s say that we are running a cart abandonment campaign and our test has a sample size of 10,000 customers. We wouldn’t send any cart abandonment emails to a random sampling of 1,000 of those customers (these guys and gals make up our control group).
Ultimately, we want to compare these two groups of customers using a common metric. The metric we use to compare is “Revenue per Customer”. After 90 days or so, we compare the value of a single customer in the holdout group vs. the value of a single customer in the marketed-to group.
Here’s an example cart abandonment holdout report from inside Rejoiner:
The above example shows that by sending this cart abandonment campaign this company is generating almost 2X the amount of revenue versus not sending it at all. (So these guys need to keep running this campaign!).
For any campaign that includes an offer or discount, we recommend running holdout tests quarterly.
Be warned: Holdout testing is not a common feature in email marketing software. But the good news is that it is an included feature with Rejoiner.
You can run hold out tests with your existing ESP just by doing some manual sorting of your list in excel and making your own randomly selected control group. For example, you could hold out a campaign for every customer with a Customer ID that ends in ‘3’ or use another filtering technique that gets at a random sampling of your list.
Setting up a hold out test in Rejoiner
For the holdout tests we run in Rejoiner, building a new test is quite straightforward. You simply choose the campaign that you’d like to start running a holdout test for, define your sample size, and launch the test. We will randomly hold out a control group (within a segment if you have one configured) and let you know when the test reaches the sample size you’ve defined:
In conclusion
Holdout tests are the only way to measure the true impact of your email program on revenue. Engagement metrics simply don’t matter when you’re trying to understand the profitability of your marketing tactics.
Holdout testing aligns your entire team around generating profitable return on marketing, not just the campaigns that generate the most clicks & orders because they contain huge discounts. They also help you answer the question, “Am I giving away margin to customers who would have purchased from us anyway?”
Here’s to understanding the true impact of your email marketing program.
Have Rejoiner run your holdout tests
We’ve helped hundreds of eCommerce companies and online retailers run holdout tests to ensure their email marketing campaigns generate the maximum amount of revenue and profit.
If your company would like access to a world-class email marketing team running your holdout tests, A/B tests and much more, then we invite you to schedule a free strategy session.
