Some people might think of the North Star as the guiding light home, taking us back to days of navigating our way home on the ships. When it comes to reports, a North Star Metric goes beyond a clickthrough rate to measure what’s working in your customer journey experience and what you need to work on. So, how can a North Star Metric help you keep your campaign goals focused, and where does A/B testing (with control groups) come into play?

A North Star Metric gives your company a focus

When it comes to campaigns, a North Star Metric might be something beyond a clickthrough rate or a read rate. This metric guides us in achieving long-term goals and long-term company improvement.

Examples of this starry-eyed metric:

• Airbnb: Number of stays booked
• Figma: Time spent designing
• Headspace: Number of meditation sessions
• Jeni’s: Number of ice cream boxes ordered (or number of Everything Bagel orders)
• Slack: Daily active users
• Spotify: Time spent listening on the platform

Ad:

A/B tests campaigns

Creating an A/B test campaign can help you learn more about what your audience likes, dislikes, where they’re at in their lives right now (like if they really want the extra guac or if they’re more apt to go for the double protein...or maybe both), and what metric to focus on.

There’s the standard A/B test where you send one email to 50% of your audience (Test A) and another email to the other 50% of your audience (Test B). You might send a different subject line to each group, a different email from name, or an entirely different message and design to see which email performs better.

For example, in e-commerce:

• 10,000 subscribers in the audience
• 3% conversion
• 300 orders
• Average order value (AOV) = $50
• Revenue = $15,000

Then, there’s the A/B test with a control group added to the mix. Adam Bianco, The Beard Club’s previous Director of Customer Marketing, dove into the value of adding a control group to A/B tests at UNSPAM 2020.

https://www.youtube.com/embed/UgIeV_kyUzA 

What’s a control group got to do with it?

Adding a control group to the A/B test campaign means that your audience will be split into three groups: Test A, Test B, and the control group. The control group will not receive any email OR the group will be excluded from receiving certain emails in an email series.

So, 45% of your audience will be in Test A receiving one email, 45% of your audience will be in Test B receiving another email, and 10% of your audience will be the control group where they do not receive an email (or the control group receives a portion of the emails in a series and are excluded from other emails in the series).

With onboarding emails, the control group might receive transactional emails (order confirmations, shipping, refer a friend), but they’ll be excluded from receiving alerts about promos, rewards, or abandoned cart reminders. In this case, the North Star Metric could be the number of friends referred, number of orders placed, or post-purchase actions (such as upgrading an order within 2 hours).

Measuring email lift

The control group helps measure the value of your email marketing efforts more accurately while connecting revenue to email marketing for non-opens. Since some open rates might be up in the air, look for different Metrics and strategies to account for the value of the emails you’re sending to customers.

Another example campaign:

• The marketable audience has 9,000 subscribers, conversion rate - 3%, number of orders - 270, AOV - $50, revenue - $13,500
• The control group has 1,000 subscribers, conversion rate - 1%, number of orders - 10, AOV - $50, revenue - $500
• The email lift = 9,000 subscribers with a 2% conversion rate, 180 orders, AOV of $50, generating $9,000 in revenue

Adding controls can help you better understand how many people converted even if they didn’t open the email. If a subscriber sees a subject line in their inbox with something straightforward and urgent (“Shop now! 24-hour sale”), that subscriber might go right to the website in their browser without opening the email so they can shop the sale ASAP. They’re included in the conversion rate without being included in the email open rate.

Measuring the customer journey experience with control groups can help you nurture relationships and understand what resonates with your audience.

How do you get your whole team to get behind finding that North Star Metric (and maybe even A/B testing with a control group)?

Look at your company’s goals from a birds-eye view. What can help the company grow in the long term? Brainstorm who your audiences would be for Test A, Test B, and the control group. Write down how they would benefit from split testing (and how the company would benefit from this).

If you’re talking to your boss about figuring out or adjusting that North Star Metric, go in with the facts, the numbers, the customer focus, and the strategy behind building and nurturing those customer relationships. At the end of the day, email is relationship designed.

Ad: