Integrate best-of-breed A/B experience testing across the app user lifecycle

We know that without shared customer intelligence across channels, marketing automation tools will fall short of delivering amazing customer experiences. As marketers shift from a reactive method of engaging with their customers to a proactive method, the Localytics and Optimizely integration will serve as an important jumping-off point for building accurate user profiles across both web and mobile marketing channels.

With this new partnership between Localytics and Optimizely, it's valuable to explore how to apply knowledge from A/B testing to optimize and personalize engagement across the app user lifecycle.

Using A/B Experience Testing in Mobile Analytics and Marketing

The first step to using A/B test data to optimize and personalize engagement is to add A/B test delivery data to app analytics and marketing. Most integrated solutions do this by:

  • Updating a custom dimension (or super property) for a device each time an A/B testing experiment is delivered
  • Having the custom dimension appear in analytics just like any other dimension (eg, device model, operating system, country)

This approach works and makes basic reporting and filtering available in analytics data, but it does so only for the latest test experiment (since each new experiment overwrites the previous one.)

A more powerful approach is to integrate A/B testing data as in-app events and user profiles. App managers already create in-app events to capture key app activities, such as making a purchase, reading an article, or submitting a registration. Adding the same A/B test data to user profiles allows app marketers to personalize push notifications, in-app messages and emails, resulting in higher app engagement and conversion.

 

The Localytics and Optimizely integration uses the event-based and profile-based approach and makes it automatic. By selecting Localytics in the Optimizely dashboard, Optimizely automatically passes A/B experiment data to Localytics. Let’s look at an example of a content app that uses this more powerful approach of integrating A/B test data as in-app events and profiles.

Measuring and analyzing the impact of A/B tests

A key goal of a content app is to drive the consumption of content, whether that content is a news article or a music video. To entice users, the app will often deliver a few experiences as part of an A/B testing experiment.  In providing these A/B tests, the content app want to answer a key question that all app managers and marketers have- which A/B test experience drives more engagement and sign-ups?

Let's say the content app focuses specifically on news articles. The app may show three different home screen experiences when a user first launches the app: (1) View the latest articles, (2) View the top articles (3) View the most shared articles. In delivering three unique experiences to different users, the app wants to know which home screen experience garners the highest engagement.

Technically speaking, each time one of these experiences is delivered to a user, Optimizely automatically creates a Localytics event called Optimizely Experiment Visited.  Each event includes two attributes - an attribute for the Optimizely experiment name: Home screen articles, and an attribute for the the name of the specific test experience delivered: latest articles, top articles or most shared articles.

To see the impact this A/B test has on future sign-ups, the news app manager can build a funnel in Localytics. In this example, the funnel has four key steps:

  1. Launching/opening the app
  2. Viewing the A/B test experience
  3. Further engaging the user by automatically sending an in-app message once he or she has read more than two articles
  4. Concluding with the goal of a new sign-up

 

Using funnels, the news app manager discovers that 114k users launched or opened the app and 0.37% of those users signed up for the app and converted. Additionally, the app manager can see the conversion rate between individual funnel steps. For example, it knows that the A/B test was delivered to 10% of new users.  Of those new users, 69% of them went on to read two or more articles and thus received the automated in-app message. Finally, 5% of users receiving the in-app message completed the sign-up process and converted.

Digging deeper into funnel step 3, the app manager can answer the original question- which A/B test experience (i.e. view the latest articles, top articles, or most shared articles) drives more engagement and sign-ups? The data shows us that each A/B test variation performed well. However, the app owner discovers that showing users a list of the most shared articles drove more people to read more than two articles.

This data can be used to help improve the conversion between funnel step 2 and funnel step 3. The data can also improve the ultimate end user experience - by engaging users in a more personal way, the app owner can attract more users to read three articles, rather than just two.

 Screen_Shot_2015-09-23_at_9.28.49_AM

Personalizing marketing to reinforce A/B tests

Using integrated analytics and personalized marketing to reinforce A/B testing is a unique capability of the Localytics and Optimizely partnership.

In the previous example, step 3 of the funnel displayed an in-app message. That message was highly targeted and sent to just a few people - those who received the home screen articles A/B test. To make the in-app message even more targeted, different messages can be sent to users based on one of the three experiment variations they received (view the latest articles, top articles, or the most shared articles.)

optimizely_audience

In the example above, an in-app message is being created to send to everyone who read at least two articles and received A/B test variation (3) view the most shared articles.

Personalized and targeted messages combined with robust analytics will help any app manager or marketer understand the best way to move users through the funnel with the ultimate goal of driving higher engagement and conversion. By bringing together our data intelligence, Localytics and Optimizely are helping marketers deliver relevant app experiences to their customers at every stage of the user lifecycle.

 

What’s Next

Over the coming months we plan to offer additional third-party integrations to help marketers share more data and refine their marketing strategy across all platforms.

 

Optimizely and Localytics Integration
 
X