How many tests can your release train afford?

Your release calendar sets a ceiling on how many tests you can run, and most teams have never counted it. A test is rarely one change. If every change rides a release, the number of tests per year is the number of releases divided by the number of changes each test needs, and that number is usually smaller than anyone on the growth side expects.

Here is how to count it, and which tests are worth moving off the train.

A test is several changes

Picture the plain version. You want to know whether a shorter upgrade screen sells better. What actually happens:

1. Ship the variant.
2. Read the first days, notice the headline is cut off on small phones.
3. Ship a fix to the variant.
4. Read again, pick a winner.
5. Ship the winner to everyone, and remove the loser.

That's three releases for one question, and we've left out the ones where the first idea is wrong and you try a second. Three is a normal number, not a bad one.

Count your trains

The arithmetic is short:

tests per year ≈ releases per year ÷ changes per test

Two things make the real number lower. A release often carries other work, so the test change may not get the next train, only the one after. And on mobile, the store's review and a staged rollout sit between "released" and "everyone has it". The hidden cost of waiting for an app release walks through pricing that wait for one change.

The count doesn't say tests are bad when they're few. It says how much your team can learn in a year, and that's worth knowing before you plan a roadmap that assumes eight.

The test isn't the slow part

When a team does run its tests, they often feel slow for another reason: reading the result takes traffic. In Miniwall's A/B guide, a result isn't called until each side has 100 visitors, and it's called when one side is very likely better, at 95% or more^[1]. Those are our tool's rules, not laws of nature, but any honest test has some version of them.

So there are two ceilings, and they're separate:

• The release ceiling, set by how often a change can reach people.
• The traffic ceiling, set by how many visitors you can split.

If your traffic ceiling is the lower one, a faster path won't help you test more. Read A/B testing with little traffic first. If the release ceiling is lower, which is what a calendar like the one above shows, the path is the thing to change. And once you're reading results, three ways an A/B test lies to you is worth ten minutes.

Which tests belong off the train

Not every test should leave it. We'd move the ones with these traits:

• The change is copy, an offer, the order of options, or what to show to whom.
• It will be tuned again, more than once.
• It doesn't depend on the device: no camera, sensors or gestures.

We'd leave on the train anything that touches checkout or payment code, and anything that has to be exactly the same for everyone for legal reasons. When not to use remote UI covers the edges.

What changes with a faster path

In Miniwall, publishing a screen needs the Editor role, and visitors get the change within about a minute^[2]. With that, the three changes in the example above happen in an afternoon instead of three releases. The test still needs traffic and time to read, so you won't run fifty a year. You'll run as many as your visitors can support instead of as many as your calendar can.

That also changes who can start one. A growth person who can publish a variant and end the test doesn't need a ticket for each step, which is a question of permissions as much as speed. Who should own remote UI and how to govern remote UI changes cover that part.

When the train is fine

If you run two tests a year and each is a big redesign that needs engineers anyway, the calendar isn't what's holding you back. The same goes if your traffic can't support more. In both cases, spend the effort elsewhere.

Next step

Take your last year. Count the tests you ran and the releases you shipped. Then take your last three tests and count how many releases each needed. If the first number is close to the second divided by the third, your calendar is your ceiling, and that's the case to bring to the next planning meeting.

Sources

1. 1.
   Miniwall docs, A/B test a paywall (plan needed, when a result can be called) (opens in a new tab)docs.miniwall.app · accessed Nov 2, 2026
2. 2.
   Miniwall docs, Publish and versions (who can publish, how fast a change goes live) (opens in a new tab)docs.miniwall.app · accessed Nov 2, 2026

Go further

• Guide: A/B test a paywall (opens in a new tab)
• Guide: Publish and versions (opens in a new tab)