How to Run A/B and Multivariate Tests in Your Push and In-App Messaging Campaigns

Do you want to understand what content and design resonates best with your app users?

Your push and in-app messaging campaigns are great testing grounds for gaining a better understanding of what copy, branding, calls-to-action, and offers work best. If you want to start testing in your mobile app, marketing campaigns are the best places to start for a few key reasons:

  • It’s easy. In the Localytics Dashboard, you can test up to 10 message versions in each push or in app campaign, without developer help or new app store submissions.

  • You can target a small audience. Some messages  flop. It’s better to flop in front of a few users than a whole crowd. In Localytics, you can target your campaigns to smaller groups of users to find out what message resonates best. Then, send the winning version to a wider audience.

  • There’s plenty to test. In-app messages are fully customizable, enabling you to test size, design, copy, color, buttons, etc. You can use the results of in-app messaging tests to inform your design and brand strategy elsewhere in your app.

In this post, we’ll cover everything you need to know to run successful A/B and multivariate tests in your push and in-app messaging campaigns.

Getting Started: Testing 101

Testing comes with its own lingo and rules. If you have been doing this for years on the web, much of this will be familiar. Here are the key terms you need to know:

Variable: The thing you are testing. In a messaging campaign, this might be copy, color, size,etc.

Version: A variant of the variable. A test can have two versions (A and B) or many (A/B/C/D/.../n).

A/B Test: Testing two versions of a single variable. The example below shows an A/B Test where the variable is button color and there are two versions: Version A (orange button) Version B (blue button). Note that besides button color, the messages are identical.

Screen Shot 2014-12-08 at 1.15.03 PM.png

Multivariate Test: Testing multiple versions of multiple variables. The example below shows a multivariate test where there are two variables: button color and copy. When testing multiple variables, it’s important to include a version for every possible combination of the variables.

Screen Shot 2014-12-03 at 10.31.00 AM.png

Testing 101 Pop Quiz!  

What kind of test is the following example?

Screen Shot 2014-12-08 at 5.07.08 PM.png

The example above represents a common point of confusion in the world of app testing. At first glance, this looks like a multivariate test, but a closer look reveals that there is only one variable: button color. But can we really call this an A/B test when there are versions C and D too?

It’s a bit of a trick question. What we’re looking at here isn’t one A/B test, but six. Because we’ve isolated one variable, we’re able to compare any two versions head to head (A/B, A/C, A/D, B/C, B/D, C/D). Running multiple A/B tests simultaneously by adding versions to a messaging campaign can expedite the testing process.

How can I test my push and in-app messages in Localytics?

Now that you’re familiar with the structure of A/B and Multivariate tests, you can formulate your own to implement in your next campaign. Here’s how:

  1. Have a goal: Testing is a means, not an end. In Localytics, you can track clicks and conversions for each message version in a campaign. Having a measurable goal, such as “increase in-app purchases” that you can track as a conversion event is the only way to understand what’s working and what’s not.

  1. Identify your variable(s): Identify a variable that could have an impact on your goal. For example, if your goal is to increase in-app purchases, you might hypothesize that different call to action (CTA) copy in your in-app messages affects conversion rates, and make that your variable.

  1. Formulate versions: Decide how many versions you are testing and what they will be. Versions of the CTA copy you might test include “Buy Now,” “Learn More,” “Go to Cart,” etc. As a starting point, you might try using copy that has worked on your website or email marketing campaigns, and iterate from there to see if users behave differently on mobile.

  1. Create your campaign: Go into the Localytics Dashboard and create a new in-app messaging campaign. In the Message Builder, define your target audience, then add your creative. In the When tab of the Message builder, you can add your versions. Once you add all desired versions, schedule your campaign and send.

Screen Shot 2014-12-08 at 3.29.27 PM.png

  1. Track performance: In Localytics, each of your campaigns has a dedicated performance screen. Go to Messaging and click into your campaign to see how your test is performing in real-time. Scroll to the bottom of the screen to see sends, clicks, and conversions for each version.

Screen Shot 2014-12-08 at 3.27.56 PM.png

  1. Analyze the results: Did your campaign have a clear winner? If yes, you can move on to testing new variables and driving further optimizations. If not, try to understand why. Try testing new variables or running a multivariate test to see if you can achieve results with other ideas.

More Resources

We have plenty of resources to help you successfully test your messaging campaigns. Check out The Ultimate Checklist for App A/B Testing and The Ultimate Guide for Mobile App A/B Testing for more best practices, ideas, and tips.

Happy optimizing!