A/B testing

A/B testing is a controlled experiment that shows two or more versions of an experience to randomly assigned groups of visitors at the same time, and compares a chosen metric between them. Random assignment and simultaneous exposure are what make the comparison causal rather than correlational, which is the entire point of the method.

How does an A/B test work?

Visitors arriving at a page are randomly assigned to a variant and held in that assignment for the duration. Each group's behavior is recorded against the same metric, and the difference between them is evaluated statistically to judge whether it is larger than chance would explain.

The two design decisions that matter most are made before the test starts. Sample size determines the smallest effect the test can reliably detect; a test that is too small will not find a real improvement, and the team will conclude the idea failed. Duration has to cover complete business cycles, because weekday and weekend visitors differ on most sites and a test that runs Tuesday to Thursday measures a population that does not exist.

Related methods sit alongside it. Multivariate tests vary several elements at once and require considerably more traffic. Split tests route visitors to different URLs. Bandit algorithms shift traffic toward the winner during the test, which is better for revenue and worse for learning.

When should you A/B test?

When the decision is reversible, the traffic supports it, and you genuinely do not know the answer.

Testing is most valuable when a change is contested internally, when the effect size is plausibly small enough that opinion cannot settle it, or when the change carries risk and you want to limit exposure. It is a poor use of time when the answer is obvious, when the page has too little traffic to conclude anything, or when the change is going in regardless of the result.

The rarely stated prerequisite is that a test requires a hypothesis. Experimentation is a verification method, not a discovery method, and it will faithfully evaluate a bad idea. Where the hypotheses come from decides how much value the testing program produces.

What invalidates an A/B test?

Four things, and all of them are common.

Stopping when the result looks good. Significance fluctuates while a test runs, so checking repeatedly and stopping at the first favorable moment manufactures wins. Fix the duration in advance and honor it.

Testing too many things at once. Running numerous tests and reporting the winners guarantees false positives. If a fifth of your tests would show significance by chance, a fifth of your winners are noise.

Sample ratio mismatch. If assignment was meant to be even and the traffic split is not, something is broken in the setup and the result cannot be trusted regardless of how good it looks.

Changing the experience mid-test, including for the control. Any edit resets the comparison, and a redirect, a campaign launch or a price change that lands mid-flight contaminates the measurement.

What are the limits of A/B testing?

It answers which, never why. A winning variant is a result. Why it won is an interpretation someone added afterwards, and it is frequently wrong, which is why findings from one test so rarely transfer to the next page.

It optimizes toward a local maximum. Testing compares options someone already imagined. It cannot surface the option nobody proposed, so a mature program tends to produce diminishing returns while the larger opportunity sits outside the tested set.

It needs traffic and patience. Most pages on most sites will never accumulate enough of either. Low-traffic and high-consideration journeys, which are often the highest value ones, are the hardest to test.

It measures the metric you chose. A variant can lift clicks and reduce completed purchases, or lift conversion and increase returns. Guardrail metrics exist for this reason and are skipped surprisingly often.

How Pulse Insights relates to this

We are adjacent to A/B testing, and we do a different job in the same program. We are not a testing platform, and if you need experimentation infrastructure, buy one.

The complementary part is the hypothesis. Testing verifies ideas; it cannot generate them, and the quality of a testing program is capped by the quality of what goes into it. Asking visitors at the moment they hesitate produces stated reasons rather than internal guesses, which is a materially better input than a brainstorm.

The other difference is what happens to an individual visitor. A test shows a variant and records the outcome; the person in the losing cell is data. Our sequence detects the friction, asks one question, and delivers a response the client's team approved in advance, in the same session. The visitor gets something out of the interaction rather than only contributing to a result.

They work well together. Use asking to find out what is actually wrong, use testing to confirm the fix is an improvement, and use asking again to understand why the winner won.

Frequently asked questions

What is A/B testing?

A controlled experiment showing two or more versions of an experience to randomly assigned visitors at the same time, to determine which produces a better result on a chosen metric.

How long should an A/B test run?

Long enough to reach the sample size decided in advance and to cover at least one full business cycle, which for most sites means a minimum of two weeks. Duration should be set before the test starts, not judged from the results.

What is statistical significance in A/B testing?

A measure of how unlikely the observed difference would be if the variants were genuinely identical. It indicates that a difference probably exists; it does not indicate that the difference is large enough to matter.

Why do most A/B tests show no difference?

Usually because the change was small, the test was underpowered, or the hypothesis was a guess. Flat results are a normal outcome of honest experimentation, and a program where everything wins is a program that is stopping early.

What is the difference between A/B testing and personalization?

A/B testing finds the version that performs best for everyone. Personalization serves different versions to different people on purpose. Testing seeks one winner; personalization assumes there is no single winner.

Related: Conversion rate optimization · Experience optimization · Website personalization · Pulse Insights vs Optimizely

Browse every term in the Pulse Insights glossary.