Multivariate & A/B Testing

The A/B Test is a technique used by marketing and communication teams to optimize the content of a campaign. It consists of testing different versions of a campaign to see which one is the best.

The indigitall platform offers a 'intelligent' version based on Machine Learning of this functionality so that this process is very simple and comfortable, so that you can use this A / B test technique regularly in your day to day day, and in this way you can optimize your campaigns based on real data of user response.

📘

Download eBook

Artificial Intelligence to attract customers. Use our algorithms to increase sales by improving communication with your audience. Explore all benefits here!

Step 1. Create an A/B Test

The creation of each notification is the same as when you create a new campaign.

On the menu, choose the Campaigns/Push A/B Test tab.

How Intelligent Send Works

The method we use is known as Multi Armed Bandit testing, MBA, inspired by the algorithm of Tompson Sampling.

While A/B tests with manual send are based on exploration, collecting data to see which is the best version and making a decision, MBAs are designed for exploitation, showing the version that seems best while testing continues .

Our algorithm progressively and automatically sends one of the versions to all devices. During the first hour we will make four shipments with all the versions and thus be able to analyze which one seems better. From the second hour every fifteen minutes and depending on the success of the versions, the one with the highest conversion rate will be sent to more users.

The longer the test lasts, the better the math model will learn and send the most successful version to more devices. Notifications will be sent in a sale of 1 to 7 days in a constant and gradual way in daytime, depending on the number of total devices and hourly filters.

A/B Test Statistics with Intelligent Send

By clicking on the statistics button in the section Campaign> Push Test A/B, you will be able to check the notifications sent and how many have been clicked by each version and device : web, android, iOS as the total.

You will get a view like the following:

On the left is the detail by campaign and in the last line the total of the campaign:

• Received: Number of devices that have received it. It can be less than those sent if there has been an error in the reception of a sent push.
• % Received: Percentage of received with respect to the total received of the campaign.
  * Clicks: Number of devices that have clicked.
• % Clicks: Percentage of clicked with respect to the total received.

On the right is the total for the campaign:

• Devices: Total number of devices
• Sent: Total and percentage of devices sent out of the total devices in the campaign.
• Clicks: Total and percentage of devices that have clicked on the total number of devices in the campaign.

When clicking on a version, a pop-up window appears with the details for each device (web, android, iOS) and the total CTR (Click Through Rate), the number of notifications clicked among those received.

Status of an A/B Test

Created tests can be in one of these states:

• Completed: Has been sent to all devices in the campaign.
• Sent: Manual send of all versions has been completed.
• In progress: The campaign is being sent by intelligent sending.
• Canceled: The test has been canceled.
• Error: An error has occurred.
• Pending: The test has been created successfully but the intelligent send has not started yet.