A/B Testing & Mobile Apps: 3 Critical Best Practices

While mobile usage is surging, app user behavior continues to be unpredictable by traditional guesswork. This trend has led marketers to look for what’s working and what isn’t in terms of app optimization and marketing efforts as quickly as possible. In the web and email space, marketers can accelerate the discovery process by A/B testing. App testing has always posed a larger challenge, but fortunately, app testing has come a long way, and we’ve identified a few critical best practices to cut down some of the trial and error to A/B testing. This post will outline the three most effective A/B testing methods for mobile apps.

Pageviews and clicks may be excellent success metrics for a/b testing on the web, but the app realm is different, so your metrics need to adapt. Mobile usage focuses on time spent in app, session length, and user behavior to qualify a successful user experience and overall, a valuable app. Each of the following methods produce results for these metrics, so if they’re unfamiliar to you, be sure to study up on the most important metrics for app measurement.


1) Testing In-App Messaging

When it comes to effective app marketing practices, in-app messaging is king. By engaging users based on certain behaviors or actions, these messages are less invasive and more relevant. While this is a great start, A/B testing of message content can quickly reveal which campaigns are most effective in meeting your goals and nudging users to desired actions more often.

For example, many news apps are constantly hunting for paid subscribers. By using in-app messaging to target a segment of users, you can test two offers side-by-side to see which is most effective at converting free to paying users.

AB testing in-app messages

After finding a winner, you can further refine your test to include messaging variables like timing, which user actions prompt the offer, and phrasing.


2) Testing Push Messaging

Push messages are the app marketing form of a tap on the shoulder. It’s natural to assume that a good app experience will always keep users coming back for more, but the truth is that user retention is a challenge when there are about a thousand things competing for someone’s attention in a given day. By sending users push notifications, you open the door for a re-engagement with your app.

The best way to build effective push messaging campaigns is to A/B test the content, frequency, and timing of the messages. For example, a retail app might introduce an A/B test for 20% off a handbag to a segment of users who have previously researched handbags on the app.

AB testing push notifications

Monitoring the results, you may find that message A is pulling 30% of users back to the app, while message B only manages a 20% conversion. Message A is the winner, right? Actually, by following the results of the campaign through to in-app purchases, users exposed to message B may drive more revenue than those exposed to message A. Going one step further in your test is the best way to capture the most actionable data and tailor your marketing efforts accordingly.


3) Testing App Content

If you’re comfortable testing web pages, then A/B testing of app content will be the easiest transition, with the most to gain in terms of functionality. While the web world focuses on alternating visual features to find more conversions, apps add the ability to test around events, or actions users take on particular screens. Events provide more qualitative user information that something like a click, because users can be grouped by events, and tracked throughout other app interactions.

AB testing app content

For example, an eCommerce app can improve its checkout process by testing buttons, icons, colors, and screen layout to see which structure funnels more users to a completed purchase, and ultimately which testing group has higher lifetime value (LTV). Every funnel has some minor or major bottleneck, and A/B testing can expedite the process of improving the user flow, with results in real time.



Keeping users engaged and driving value for your app is no easy feat, but you can rapidly improve both the app experience and the impact of your app marketing efforts through strategic, iterative A/B testing. To dive deeper into best practices and case studies for A/Btesting in the mobile space, check out the Ultimate Guide to Mobile App A/B Testing.

What’s the first a/b test you want to run for your app?

Localytics Ultimate Guide to Mobile APP A/B Testing