Your A/B Test May Be Lying to You

Stopping an A/B test the moment the result “looks good” has a name: peeking. The practice seems harmless, but it breaks the experiment's statistical guarantees. In a simulation with 10,000 tests, it took the false positive rate from 5% to almost 20%.

A/B tests are widely used to compare independent populations, and the logic seems simple: measure whether there is a statistically significant difference in a metric between two groups, A and B. In digital marketing, the most common use is deciding which creative performs better on a given metric (such as click-through rate) and, based on that, directing spend only to the more promising version. Except that two details of the statistics behind the test usually go unnoticed. This article shows both, with numbers.

• How the frequentist A/B test works
• What the test actually answers
• Peeking: the mistake of checking early
• The simulation: the jump in false positives
• How to test without falling into the trap

How the frequentist A/B test works

Let's take an example with two ads:

• Version A: the baseline
• Version B: identical to A, but with a red button

In a frequentist A/B test, the process usually follows three steps:

1. The audience is randomly split between the two groups.
2. Impressions and clicks are measured for each version.
3. Based on the data, you assess whether it is possible to reject the hypothesis that A and B have the same click-through rate, adopting, for example, a 95% confidence level.

The step by step of setting up, measuring and reading an experiment is in our article on A/B testing in practice. Here, the focus is different: what the result actually means and how it can deceive.

What the test actually answers

An important point belongs here: the test does not directly answer the question “which ad is better”. What it answers is subtler and less interesting.

The test starts from the hypothesis that there is no real difference between the two ads. It is the so-called null hypothesis:

H₀: p_A = p_B

In it, p_A and p_B are the true click-through rates of versions A and B. Assuming this hypothesis, the test calculates how unlikely it would be to observe the difference we are seeing in the data through random variation alone. If the observed result is unlikely enough under H₀ (a probability below 5%, for example), we say it is statistically significant.

Note what that implies: the 95% confidence level already accepts, by construction, 5% false positives. In 1 out of every 20 tests of identical ads, we expect to declare a winner that does not exist. That is the agreed price. The problem is when the decision process inflates that price without anyone noticing.

Peeking: the mistake of checking early

Now comes the second problem, and perhaps the most serious one. It is common to see teams tracking results throughout the experiment and deciding to stop the test when the result “looks good”. This practice, however intuitive, breaks the test's statistical guarantees and drastically raises the risk of false positives.

The reason: the 5% guarantee holds for a single analysis, done on a sample defined before the start. When the team peeks several times and stops the test at the first favorable peek, each peek becomes a new chance for randomness to cross the significance line. The difference between the groups oscillates throughout the experiment out of pure sampling variation, and stopping at the peak of the oscillation is selecting the noise, not the signal.

The simulation: the jump in false positives

We simulated this scenario to visualize the real impact:

• We ran 10,000 tests with identical A and B ads, both with a true click-through rate of 10%.
• Each test was “peeked at” 10 times before being finalized.
• Result: the false positive rate jumped from 5% (the expected value) to almost 20%.

In other words, even with no real difference between the groups, the test declared one of the versions superior almost 20% of the time. In practice, this considerably increases the risk of wrong decisions based on nothing but random variations in the data: budget reallocated, creative paused and a “learning” logged about an effect that never existed.

The simulation code is in a public Colab notebook, for anyone who wants to reproduce the numbers or vary the parameters.

How to test without falling into the trap

Within the frequentist framework, the protection is discipline: calculate the sample size before starting, run the test until it reaches that size and only then look at significance. Peeking along the way, without statistical correction, invalidates the math.

At Marktech, we use more flexible methods for A/B testing, which allow continuous analysis of the results without compromising the experiment's mathematical validity. They also offer more direct answers to the practical question that matters: which ad is more likely to be superior?

The same rigor that protects a creative test applies to bigger media decisions, such as estimating each channel's effect on results: that is the subject of our article on Marketing Mix Modeling with Meridian.

Before ending your next test, it is worth asking: was the sample defined before the start? How many times has the result been peeked at? And, looking back, have you fallen into this trap before?