A/B Testing

Introduction

In this guide, you will learn how to use the Odicci A/B Testing feature with any of your experiences.

Note: The A/B Testing feature is a premium feature that is currently in beta. To learn more about how to activate this feature on your account, please contact Odicci Customer Success.

What is an Odicci A/B Test Experience?

An A/B Test experience is an experience that allows you to run two variations of your experience simultaneously at first, allowing you to monitor how each variation performs. After Odicci has determined which variation has performed better, based on the criteria you want to use, the better performing variation (i.e., "winner") is determined and will be the variation that is shown to your customers going forward.  

How can I create an A/B Test Experience?

To create an A/B Test Experience:

  1. Create a new experience or open an experience that has not been published.
  2. In the "Studio" screen, click the "A/B Test" icon at the top-right corner of the canvas, next to the experience configuration icon.
  3. A notification is displayed informing you that activating the A/B Test for the experience cannot be undone. Click "OK" to proceed.
  4. Once you activate the A/B Test feature for your account, a drop-down menu appears at the top-left corner of the screen where you can navigate between both versions (A and B) of your experience.
  5. Make the desired changes to the variations in your experience:
    • on the "Setup" screen, you can update the names of both versions if you want, or you can keep the default names.
    • on the "Prizes" screen, you can apply different prizes or make changes to existing prizes on both variations.
    • on the "Studio" screen, you can make any changes you want to the experience content for both variations.
  6. Once you've completed making the changes to your variations, go to the "Confirm" screen of the experience and from the "A/B Test" section, select how you want the winning variation to be determined from the following options:
    •  "Using Click Through Rate (CTA)": Selecting the click through rate allows you to configure how the winning variation will be determined by one of the following two methods:
      • the variation with the highest click through rate at the end of a certain number of days will be determined to be the winning variation
        or;
      • the variation with the highest click through rate once a number of entries is reached will be determined to be the winning variation.
    • "Using Opt In Rate": Selecting the opt in rate allows you to configure how the winning variation will be determined by one of the following two methods:
      • the variation with the highest opt in rate at the end of a certain number of days will be determined to be the winning variation
        or;
      • the variation with the highest opt in rate once a number of entries is reached will be determined to be the winning variation.
    • "Using Completion Rate": Selecting the completion rate allows you to configure how the winning variation will be determined by one of the following two methods:
      • the variation with the highest click through rate at the end of a certain number of days will be determined to be the winning variation
        or;
      • the variation with the highest click through rate once a number of entries is reached will be determined to be the winning variation.
    • "Manually": Selecting the manual method will allow the experience to run until you return to the "Confirm" screen of the experience to select the winning variation.
  7. Once you have configured how the winning variation will be determined, you can Publish the experience.  (Note: Once the configuration of how the winner will be determined, you can no longer change the configuration of the winner determination)
  8. Once you've published your experience, you can monitor the performance of the experience in the "Reports" section:
    • The "A/B Dashboard" screen which is divided into the following sections:
      • The aggregate performance of each variation is displayed in a chart for the following KPIs: Impressions, Users Engaged, Completed Entries, Opt in Rate, and CTA Click Through's
      • The date graph where you can see the performance of both variations shown over time for the following KPIs: Impressions, Users Engaged, Completed Entries, Opt in Rate, and CTA Click Through's