Key takeaways
  • A/B tests are the gold standard for causal evidence, and the most misused tool in product.
  • Mind statistical power and sample size; don't "peek" early, and respect the 5% false-positive rate.
  • "Always running tests" is cargo-cult; some tests aren't worth running.

An A/B test is the most powerful research tool a product team owns, and the one most likely to be misused without anyone noticing. Used properly, it produces causal evidence — the kind of evidence that lets you say the new button made activation go up, not activation went up while the new button happened to be live. That distinction is the entire point. Almost every other form of product measurement gives you correlation. A well-designed A/B test gives you cause.

Used badly, and most teams do use them badly, A/B tests produce confident-looking numbers attached to conclusions that aren't supported by the data. Teams ship "winning" variants that won by chance. Teams kill features that would have worked because the test was underpowered. Teams check the dashboard every morning and call the experiment when they like what they see. The maths is unforgiving, and the costs of getting it wrong compound across every decision the team made on bad evidence.

What an A/B test is, and why it works.

An A/B test takes the population of users meeting some criterion, randomly assigns them to two groups (or more), exposes each group to a different version of the product, and compares an outcome metric between the groups. The mechanism that makes the test valuable is the randomisation. Because group assignment is random, the two groups are statistically equivalent on every variable, known and unknown, except the one you're testing. Any systematic difference in outcome is then attributable to the difference in treatment.

This is the same logic as a randomised controlled trial in medicine. It's the cleanest tool we have for establishing causation, and the rest of the product evidence stack (analytics, surveys, user interviews) gives you correlation or self-report at best. Ron Kohavi's work on experimentation at scale is the standard reference. Read it if your team runs more than a handful of tests a quarter.

The architecture of an A/B test.

Diagram · The architecture of a properly designed A/B test
The architecture of a properly designed A/B test — traffic splitter, two variants, metrics, significance test A flow chart showing the structure of an A/B test. At the top a wide rectangle labelled Traffic Splitter randomly assigns incoming users to one of two arms. Two arrows flow downward to two parallel boxes: Variant A on the left in dark ink, and Variant B on the right in marigold. Both variants flow down into a Metrics box that captures the outcome measurements. Below that, a marigold diamond labelled Significance Test evaluates the result against the agreed thresholds: statistical power of eighty per cent and alpha of five per cent. Two outputs leave the diamond: No Win on the left in cream, and Win on the right in marigold. Side annotations on the right note power equals eighty per cent and alpha equals five per cent. Traffic Splitter CONTROL Variant A current experience TREATMENT Variant B new experience Metrics — capture outcome Significance Test No Win Win power = 80% α = 5% Randomisation is what makes the test causal — not the dashboard.
The structure looks simple. Every error in interpretation comes from skipping one of the four boxes — typically the significance test, replaced by "the dashboard looks good."

Statistical power, and the sample-size question.

The single number that decides whether your test is worth running is the sample size, and it's set by three inputs: the baseline rate of the metric you're measuring, the smallest effect size you'd care about, and the statistical power you want to achieve (conventionally 80%). Plug those three into a sample-size calculator (Evan Miller's is the canonical one) and you get the number of users per arm the test needs.

That number is often much, much larger than teams expect. To detect a 2% relative lift on a baseline 10% conversion rate at 80% power, you need roughly 30,000 users per arm. If your monthly traffic is 5,000 users total, the test will take three months to reach significance, and three months later, the world has changed, seasonality has crept in, and the question has moved on. The honest answer is that the test isn't worth running. Make the decision on smaller evidence and a strong prior, and accept that.

The peeking problem.

The most common and most damaging mistake is "peeking" at the test before it has reached its planned sample size. The standard significance threshold (α = 0.05) is calibrated to a single test at a single moment. Every time you look at the result early, you have to recompute against the new sample, and the probability of seeing a false-positive purely from sampling noise multiplies. A test that's "p < 0.05" on day three of a fourteen-day run is not actually significant in any defensible sense. It's noise dressed up as a result.

The fix is mechanical: agree the sample size before the test starts, write it down, and don't look at the result until it lands. If you can't help looking, use a sequential testing framework that's mathematically robust to peeking. Group sequential designs and Bayesian methods both exist for this. They are slightly more complex to set up and far more honest in their outputs.

The 5% false-positive rate.

Even a perfectly run test, with the agreed sample size and no peeking, will report a "significant" result 5% of the time when there's actually no effect. That's what α = 0.05 means: the test is calibrated to admit a 1-in-20 false-positive rate. Run twenty tests this quarter where the truth is "no effect," and you'll ship one variant that wasn't actually better. Over time and at scale, this matters. A team running fifty tests a quarter and shipping every "winner" is shipping two or three false wins every quarter without knowing.

The countermeasures are sober: don't ship a winner if the lift is implausibly large for the change made (a 30% lift from a button colour change should make you suspicious, not happy); replicate important results before treating them as the basis for strategy; pre-register your hypothesis so you can't quietly rewrite it after the fact to fit the result you got.

The "always running tests" cargo-cult.

Some teams treat the number of experiments per quarter as a metric in itself. Leadership cite "we ran 120 tests this quarter" as evidence of rigour. It is not. If the average test is underpowered, the average test is noise, and a high test count is just noise at scale. A team running five well-powered tests on important questions learns more than a team running a hundred underpowered tests on button copy.

A/B testing isn't a way to skip thinking. It's a way to confirm thinking.

When a test isn't worth running.

  • Low traffic. Sample-size maths says the test would take months. The decision can't wait. Make the call on judgement and prior evidence.
  • Qualitative question. "Why didn't users get the value?" is not an A/B test. It's an interview. Five well-conducted user interviews will tell you more than a numerical lift.
  • Effect size unknowable in advance. If you can't make a reasonable guess at the minimum detectable effect, the test will be either over- or under-powered, and either is wasted.
  • Strategic, not tactical. "Should we enter market X" is not an A/B test. It's a business decision. A/B tests are good at button colour. They are useless at market choice.
  • Already known. The variant under test is so obviously better that running the test is just confirmation bias on a calendar. Ship it.

Sequential vs fixed-horizon.

A fixed-horizon test is what most product teams run by default: agree a sample size up front, run until you hit it, then look at the result. A sequential test is one that's designed from the start to allow honest looks at the data while the test is running, by adjusting the significance threshold to compensate for the multiple looks. Sequential tests are more efficient (you can stop early if the result is overwhelming) and far more honest about how teams actually want to use the data. If your team can't stop themselves peeking, run sequential tests. The maths is harder. The conclusions are sounder.

CUPED, briefly.

Controlled-experiment Using Pre-Experiment Data, or CUPED, is a variance-reduction technique that lets you reach significance with smaller samples by using each user's pre-experiment behaviour as a control. If your team has the analytical capacity, CUPED can cut required sample sizes by 30–50%. Microsoft, Netflix and Booking.com all use variants of it as default. For smaller teams it's a stretch, but it's worth knowing the technique exists once you've outgrown the basics.

How we use this at Product Pieces.

We use A/B tests as a confirmation tool, not a discovery tool. Discovery happens upstream: interviews, prototypes, user research (see N°21 and N°22). By the time something reaches an A/B test, the team should have a strong prior that the new variant will win, and the test is there to size the win and rule out the case that the prior was wrong. Tests run as fishing expeditions ("let's try ten button copies and see what works") almost always produce noise, and the noise gets shipped as if it were signal.

The other thing we insist on is a clear stop condition. Sample size agreed up front. Date the test will end. No peeking. If the result isn't significant at the planned end, the variant doesn't ship — extending the test until significance appears is the most common way teams convince themselves they have evidence they don't.

Next issue: N°24 — RICE. A scoring system, not a decision system.