Three ways an A/B test lies to you, shown by simulation

A test can report a winner when the two versions are exactly the same. How often it does depends less on your statistics library than on how you behave while the test runs: how often you look, how many versions you try, and how many ways you cut the data afterwards. In the simulation below, an honest-looking 5% error rate turned into 27% for someone who looked every day.

Everything here comes from one script you can run. The inputs are numbers we picked (a 4% purchase rate, 36 visitors per version per day, 28 days), not data from anyone's product. Both versions get the same true rate, so every "winner" the script finds is noise. This follows from A/B testing with little traffic, which covers sizing a test before you start. This post is about what happens after it begins.

The script

It's plain Node with no packages, and it uses a small seeded random generator so the output is the same on every run. We ran it on Node 22 with seed 1.

// Illustration only, not any product's API. Run: node peeking.mjs
// A/A simulation: both versions have the SAME true purchase rate, so every "winner" is noise.

const RATE = 0.04;       // true rate, same for both versions (example)
const PER_DAY = 36;      // visitors per version per day (example: ~2,000 a month in total)
const DAYS = 28;
const RUNS = 2000;
const SEED = 1;

// Small seeded random generator (mulberry32), so the output is the same every run.
function rng(seed) {
  let a = seed >>> 0;
  return () => {
    a = (a + 0x6d2b79f5) >>> 0;
    let t = a;
    t = Math.imul(t ^ (t >>> 15), t | 1);
    t ^= t + Math.imul(t ^ (t >>> 7), t | 61);
    return ((t ^ (t >>> 14)) >>> 0) / 4294967296;
  };
}

// Two-proportion z statistic with a pooled standard error. Returns 0 when it is undefined.
function z(salesA, viewsA, salesB, viewsB) {
  const pooled = (salesA + salesB) / (viewsA + viewsB);
  const se = Math.sqrt(pooled * (1 - pooled) * (1 / viewsA + 1 / viewsB));
  return se === 0 ? 0 : (salesB / viewsB - salesA / viewsA) / se;
}

const Z_05 = 1.959964;   // two-sided 5%
const Z_BONF5 = 2.575829; // two-sided 1% (0.05 / 5 per comparison)

// 1) Looking more often
const rand = rng(SEED);
const policies = { "look once, on day 28": [28], "look every week": [7, 14, 21, 28], "look every day": Array.from({ length: DAYS }, (_, i) => i + 1) };
const hits = Object.fromEntries(Object.keys(policies).map((k) => [k, 0]));

for (let run = 0; run < RUNS; run++) {
  let a = 0, b = 0;
  const bySalesDay = [];
  for (let d = 1; d <= DAYS; d++) {
    for (let i = 0; i < PER_DAY; i++) {
      if (rand() < RATE) a++;
      if (rand() < RATE) b++;
    }
    bySalesDay[d] = [a, b];
  }
  for (const [name, days] of Object.entries(policies)) {
    // "stop at the first look that shows a winner"
    if (days.some((d) => Math.abs(z(bySalesDay[d][0], d * PER_DAY, bySalesDay[d][1], d * PER_DAY)) > Z_05)) hits[name]++;
  }
}
console.log("False winners in A/A tests (both versions identical):");
for (const [name, n] of Object.entries(hits)) console.log(`  ${name}: ${((n / RUNS) * 100).toFixed(1)}%`);

// 2) Many versions at once: 5 new versions against one control, one look on day 28
const rand2 = rng(SEED + 1);
const N = DAYS * PER_DAY;
const draw = () => { let s = 0; for (let i = 0; i < N; i++) if (rand2() < RATE) s++; return s; };
let any05 = 0, anyBonf = 0;
for (let run = 0; run < RUNS; run++) {
  const control = draw();
  let found05 = false, foundBonf = false;
  for (let v = 0; v < 5; v++) {
    const zz = Math.abs(z(control, N, draw(), N));
    if (zz > Z_05) found05 = true;
    if (zz > Z_BONF5) foundBonf = true;
  }
  if (found05) any05++;
  if (foundBonf) anyBonf++;
}
console.log("\n5 identical new versions vs one control, one look on day 28:");
console.log(`  at least one 'winner' at 5% per comparison: ${((any05 / RUNS) * 100).toFixed(1)}%`);
console.log(`  at least one 'winner' at 1% per comparison (0.05 / 5): ${((anyBonf / RUNS) * 100).toFixed(1)}%`);

// 3) Slicing after the fact: split each version's visitors into 8 groups, check each group
const rand3 = rng(SEED + 2);
const GROUPS = 8, PER_GROUP = Math.floor(N / GROUPS);
let anySlice = 0;
for (let run = 0; run < RUNS; run++) {
  let found = false;
  for (let g = 0; g < GROUPS; g++) {
    let a = 0, b = 0;
    for (let i = 0; i < PER_GROUP; i++) { if (rand3() < RATE) a++; if (rand3() < RATE) b++; }
    if (Math.abs(z(a, PER_GROUP, b, PER_GROUP)) > Z_05) found = true;
  }
  if (found) anySlice++;
}
console.log(`\nSlice into ${GROUPS} groups of ${PER_GROUP} per version, look at each: at least one 'winner' in ${((anySlice / RUNS) * 100).toFixed(1)}% of tests`);

Each scenario below is 2,000 fake tests. "Winner" means the usual two-sided 5% test on the two rates came out significant.

1. Looking early

The first part has three people run the same 28-day test. One looks once, on day 28. One looks every Monday and stops if the test shows a winner. One looks every day and does the same.

The 5.5% for one look is close to the 5% the test promises. (With 2,000 runs, a result half a point either side of 5% is ordinary wobble.) The other two rows are not close. Each extra look is another chance for a lucky streak to cross the line, and a daily checker has 28 of them. Stopping at the first significant look, which is the tempting thing to do when you see something good, turns "one in twenty" into more than one in four.

Notice that nothing about the data changed. Same visitors, same rate. Only the habit did.

What we'd do: write the end date down before you start and don't act on anything before it. Looking is fine. Deciding on what you saw is the problem. There are methods built for checking repeatedly and stopping early while keeping the error rate honest, usually called sequential testing. We haven't covered them here and we haven't opened a source to cite for any particular one, so read about them before trusting one that your tool offers.

2. Trying many versions

Now one control and five new versions, all identical, looked at once on day 28. If each comparison is judged at 5%, the chance that at least one of the five looks like a winner is not 5%.

Dividing the 5% by the number of comparisons is called a Bonferroni correction. It brought the false-winner rate back near 5% here. It's the conservative option, not the only one: it asks a lot of each comparison, so a real improvement is harder to see. We use it because you can do it in your head. Other corrections exist, and which is best depends on what you're testing.

The cheaper fix is to test fewer things at once. Five versions at the same traffic means each gets a fifth of the people, which on a small audience is its own problem.

3. Cutting the data afterwards

Say the test ends with no clear result, and someone says, "Let's see how it did for people who arrived on a weekend, or on mobile, or from search." In the third part we split each version's visitors into 8 groups of 126 and checked each group.

In 30.2% of the simulated tests, at least one group showed a "winner". Again, the versions were identical. Eight slices is eight chances, and the post-hoc ones are the worst kind, because you chose which slices to look at after you'd seen the numbers.

Slices aren't off limits. Slicing is a good way to find a question for the next test. It's a bad way to answer the current one. Write the segments you care about before the test, keep their number small, and treat a surprise as a hypothesis.

What the simulation doesn't tell you

It's one setup. A 4% rate, an even split and a plain z-test are examples, and a different rate or sample size would give different percentages. The pattern (more looks, more versions, more slices, more false winners) is the part we'd rely on, not the exact figures. We also haven't simulated the other known traps, such as people reacting to anything new for the first week, or weekday and weekend visitors behaving differently. We know about them, we haven't measured them, so we won't put numbers on them.

And a test that finds nothing isn't a failed test. "No difference we could see at this size" is an answer, and often a good enough reason to pick the version that's cheaper to keep.

If you do one thing, copy the script, change RATE and PER_DAY to your own, and watch what your habits would have cost you.