A/B Testing: A Powerful Optimization Tool
A/B testing is one of the most effective ways to increase conversions. See when to use it, how to measure significance, and what to do with the result.
A/B testing is one of the most effective ways to increase conversions on your website.
In this article, we will explain how A/B tests work and how you can put them to use in your business.
- When should you use an A/B test?
- How do you run an A/B test?
- What should you do with the A/B test result?
- Tip #1: Stop the test!
- Tip #2: Keep a repository of A/B tests.
- Tip #3: Run more tests!
When should you use an A/B test?
When we run an A/B test, we want to understand some behavior of a group.
With that information, we can optimize a metric that matters to the business, such as visits, clicks, lead generation, or sales. You can test almost anything: an ad, an email, your pricing and, above all, the landing page experience.
To illustrate, let me tell you about João, who runs an online shirt store. João has the important task of deciding which shirt goes on sale in his online store.
Since he does not know which choice will bring better sales results, he decides to run a test: putting two different versions of his site live, where half of the users are sent to variation A and the other half to variation B.
How do you run an A/B test?
After creating variations A and B for your tests and measuring their results (metrics such as clicks, sales, or leads), you need to calculate the statistical confidence level.
That confidence level tells you whether or not there is statistical significance to declare one of the test variations the winner. And significance depends on sample size: with few visits and conversions, no difference can be trusted, however large it looks.
For example, let's imagine João ended the test with the following data:
| Variation | Clicks | Sales | Conversion rate |
|---|---|---|---|
| A | 1,000 | 30 | 3% |
| B | 1,000 | 50 | 5% |
According to the data, variation B achieved a higher conversion rate than variation A.
To calculate the confidence level of this experiment, we can use an online A/B test calculator. Entering this data into that kind of calculator, we get a 97% confidence level for the experiment, above the usual 95% cutoff.
In other words, there is enough statistical confidence to declare variation B the winner.
What should you do with the A/B test result?
After running the A/B test, a few actions are essential to get the most out of your experiment:
Tip #1: Stop the test!
Once you have statistical confidence and know which variation is best, it is important to take the losing variation down.
The losing variation is the one with lower performance, with statistical confidence. In João's case, for example, variation A was the loser of the test.
The opposite also holds: do not end the test too soon. Stopping early, with a small sample, is the fastest way to crown a false winner.
Tip #2: Keep a repository of A/B tests.
Beyond being a powerful optimization tool, the test gives you a relevant piece of business knowledge that should be stored.
In the example of João's store, he learned that his audience responds better to sales on the yellow shirt.
If João runs a sale next month, he will pick the yellow shirt, putting to work the business knowledge he gained from this A/B test.
Tip #3: Run more tests!
It is important to keep running A/B tests to continuously optimize your site and your results. The same reasoning applies to ads: testing different headlines and descriptions is the shortest path to improving ad relevance in Google Ads. Marktech maintains free tools for advertisers: the A/B test calculator is one of them.
Want help designing and running your tests? Talk to us.
Want these results in your campaigns?
Talk to a Marktech consultant and get a free diagnosis of your media operation.
Talk to a consultant
