A/B testing with little traffic: what you can still learn

If your paywall is seen by a couple of thousand people a month, an A/B test can tell you that a new version is much better or much worse. It usually can't tell you that it is 10% better. So size the test before you run it, and if the size is out of reach, test a bigger change or don't test that thing at all.

That's the short answer. The numbers below are examples we picked to be easy to follow, not data from anyone's product. The code further down reproduces all of them, so you can swap in your own.

Small differences need a lot of people

Imagine a paywall that 2,000 people see each month, and 4% of them buy. That's an example, not a benchmark. You change the headline and hope for a lift. How many people have to see each version before a test can reliably notice it?

The answer depends on how big the difference is, and it depends on it harshly. The gap between the two rates enters the formula squared. Halve the lift you're hoping to catch and you need roughly four times the people.

Two of the terms in that caption deserve a plain reading. Significance of 5% means that if the two versions were truly identical, the test would call a winner by accident about one time in twenty. Power of 80% means that if the true lift were exactly the one in the row, the test would catch it about four times out of five. The fifth time, a real win goes unseen. Statistics libraries ask for the same inputs: statsmodels describes the significance level as "the probability of a type I error" and power as "one minus the probability of a type II error"^[1].

Read the last row again. A 10% lift on a 4% rate is a change from 40 sales per thousand views to 44. At 2,000 views a month, that's a test of more than three years. Nobody runs it, and nothing else about the product would stay still for three years anyway.

Reading a small result honestly

Say you run it anyway, with 400 people on each side. Version A gets 16 sales and version B gets 24. That's 4% against 6%, a 50% jump, and it's tempting to ship B and tell the team.

Look at the interval around the gap instead. With the same approximation as the sample-size code, the 95% interval for the difference runs from about -1.0 to +5.0 percentage points. B could be a little worse. B could be a lot better. The data fits both, because 16 and 24 are numbers small enough that a handful of lucky or unlucky days can produce the whole gap.

The interval is the useful thing to bring to a decision. "B won" is a sentence. "B is somewhere between slightly worse and much better" is what the data actually said. (For counts this small, the simple formula we use is rough, and there are better interval methods. We kept it simple so you can follow every step.)

The code

It's plain Node, no packages, and it needs Node 18 or later. The two constants are the standard normal values for a two-sided 5% test and 80% power.

// Illustration only, not any product's API. Run: node sample-size.mjs
const Z_ALPHA = 1.959964; // two-sided 5% significance
const Z_BETA = 0.841621;  // 80% power

// Visitors needed in each version to tell rate p1 from rate p2 (normal approximation).
export function perVersion(p1, p2) {
  const pbar = (p1 + p2) / 2;
  const a = Z_ALPHA * Math.sqrt(2 * pbar * (1 - pbar));
  const b = Z_BETA * Math.sqrt(p1 * (1 - p1) + p2 * (1 - p2));
  return Math.ceil((a + b) ** 2 / (p2 - p1) ** 2);
}

// 95% interval for (B minus A), in percentage points. Same approximation.
export function diffInterval(salesA, viewsA, salesB, viewsB) {
  const pA = salesA / viewsA;
  const pB = salesB / viewsB;
  const se = Math.sqrt((pA * (1 - pA)) / viewsA + (pB * (1 - pB)) / viewsB);
  const d = pB - pA;
  return [d - Z_ALPHA * se, d + Z_ALPHA * se].map((x) => +(x * 100).toFixed(1));
}

const base = 0.04;
for (const lift of [0.5, 0.25, 0.1]) {
  console.log(`+${lift * 100}%: ${perVersion(base, base * (1 + lift))} per version`);
}
console.log(diffInterval(16, 400, 24, 400)); // [ -1, 5 ]

Change base and the lifts to your own rates. If your estimate of the starting rate is shaky, run the numbers for a couple of values and plan for the worse one. Other calculators may use a slightly different formula, so expect small differences and check yours against this one.

Decide four things before you start

Small tests get ruined at the start, before any data arrives. We'd write these down first, in a place the whole team can see:

1. One change. If you change the headline, the price display and the button, a win tells you nothing about which one did it.
2. One number that decides. The one you actually care about, usually purchases per view. Other numbers can sit beside it as a check, but pick the decider now. When several numbers are on the table afterwards, one of them will always look good.
3. When it ends. A number of visitors, or a date that's a whole number of weeks, written down in advance. Every time you look and consider stopping, you give noise another chance to look like a win. We put a number on that in reading experiment results without fooling yourself.
4. What you'll do for each outcome. Including the likely one, "no clear difference". If B is cheaper to maintain, or you like it better, an inconclusive result is a fine reason to ship it. It means you didn't find a difference at this size, not that the test failed.

When the sample is out of reach

At low traffic you're usually looking at the first two rows of the table, or worse. Here is what we'd consider instead, roughly in order.

Test a bigger change. The +50% row is reachable; the +10% row isn't. So spend the test on something that could plausibly move the number by that much: a different offer, a different moment to ask, a different plan shown first. A new button colour won't do it, and you'd have proven that only after months.

Count something that happens more often. Clicks on a plan happen more than purchases, so a test on clicks finishes sooner. It comes with a real risk: a version can win clicks and lose sales, for instance by promising something the checkout doesn't deliver. If you use clicks, treat the result as a hint and look at purchases once you have them.

Use all your traffic. A test on one country, or one plan, or one entry point shrinks your 2,000 views to a fraction. Where the change makes sense for everyone, show it to everyone.

Compare before and after, with eyes open. Ship the change, note the date, compare the weeks either side. This is weak, because anything else that moved in those weeks (a release, a campaign, a holiday) lands in the comparison. If you do it, change one thing at a time, write down everything else that happened, and call the outcome a hint.

Talk to people. A handful of conversations with people who saw the screen and didn't buy will often show a problem no test of this size would find. Ask what they thought they were getting. This is our opinion, and it costs an afternoon.

Just decide. If the change is easy to undo and the downside is small, a test can cost more than being wrong. Ship it, watch the purchases, and roll it back if it looks bad. You've given up the claim that you know it worked.

When this advice doesn't apply

If you have tens of thousands of views a month, the +10% row comes within reach in a few months, and a proper test earns its keep. And if a result would be expensive to get wrong, such as a price change you can't walk back, then the size of the sample is the price of being sure, and "just decide" isn't the right call.

One more limit: everything above is about a single comparison of two rates. Several versions at once, or several numbers at once, need bigger samples and more care. We simulate what goes wrong in the post on reading results.