Why a paywall test can win on conversion and lose

If you test two prices and judge them by how many people buy, the cheaper one will usually win. That tells you very little. What you want to know is whether you end up with more money, from people who stay, and conversion can't say. Pick one number that decides, before the test starts, and make it something closer to revenue than to clicks.

The numbers below are an example we made up to be easy to check by hand. They aren't data from anyone's product.

A cheaper price that wins and still loses

Imagine 10,000 people see each version of a paywall. Version A charges $10 a month. Version B charges $6. Say 4.0% buy A and 5.5% buy B. Say also that after the first month 80% of A's subscribers renew and 70% of B's do. Those retention figures are made up too, but "people who pay less leave sooner" is a pattern worth checking in your own data rather than assuming away.

B's conversion is 37.5% higher (550 against 400). B's revenue per visitor in the first month is 17.5% lower. After the second month the gap is wider: about 22% lower. A report that says "B converts better" is true and points you at the wrong version.

One more number is useful when you're deciding what a lower price has to deliver. To match A's first-month revenue per visitor, B at $6 needs 4,000 / 6, about 667 subscribers, which is 6.67% conversion. So a cheaper price has to buy you a lot more buyers just to break even, before retention gets a vote.

None of this means the higher price always wins. If B's retention matched A's and its conversion were 7%, B would come out ahead. The point is only that conversion can't tell you which case you're in.

One number decides, written down first

We'd pick one deciding metric before the test and put it somewhere the team can see. For a paywall it's usually revenue per visitor who saw the screen, over a period you chose in advance. Not purchases per view, unless the prices are the same in both versions and you're testing something else, like a headline.

The reason to fix it early is the one we gave in reading experiment results without fooling yourself: with several numbers on the table afterwards, one of them always looks good, and it's tempting to call that one the result.

Revenue per visitor has a cost. Most visitors pay nothing and a few pay something, so the number is noisy, and a small audience will struggle to see anything but a big difference. A/B testing with little traffic works through how big. If that's out of reach, conversion can still be a stopgap, as long as both versions show the same price and you say so.

Guardrails: the numbers that must not get worse

A guardrail is a metric you don't try to improve in the test but watch, so a win on the main number doesn't hide damage somewhere else. We'd start with these:

• Early cancellations. People who subscribe and cancel within the first days, or end a trial without paying. A version that sells a promise the product doesn't keep shows up here first.
• Refunds and chargebacks. They arrive late, but they're the most direct sign that people felt misled.
• Support contacts about billing. If you can count them by version, count them.

Pick two or three. Each extra guardrail is one more number that can wobble by chance, which is the multiple-numbers problem again, so a guardrail needs a rule you wrote earlier: "if early cancellations rise by more than X, we don't ship, whatever revenue says." We make no claim about what X should be. It depends on what a cancellation costs you.

Renewals come late, so decide how long to wait

The first payment shows up at once. The second one, the one that tells you whether people stay, arrives a billing period later. For a monthly plan that's at least a month after the last person entered the test. For an annual plan it's a year, and nobody waits a year.

You have to decide, before starting, how much of the future you'll count. Some options:

1. Wait for the first renewal. Slow but honest for monthly plans. Plan for the test to end a month after the last visitor joins.
2. Use an early signal. For example, still subscribed after the first week, or trial started and not cancelled. Faster, but it's a proxy, and you should say so when you report it.
3. Decide on the first payment, and check retention later. Reasonable when the change is easy to undo. Ship the winner, keep watching the cohort, and be ready to roll back.

Whichever you choose, write it down along with what you'll do if early and late data disagree. We'd lean towards the slower number for anything you can't easily reverse, like a price change, and the faster one for copy and layout.

Where Miniwall fits

Sources

1. 1.
   Miniwall docs, Events and privacy (events sent, and what is never sent: prices, amounts) (opens in a new tab)docs.miniwall.app · accessed Oct 26, 2026

Go further

• Guide: Events and privacy (opens in a new tab)