Skip to content

How to tell if remote UI is paying off

Build, buy & decideOpinion4 min readMiniwall team

Part of the series: Ways to change a live app without shipping a release

Measure four things: how long a change takes from decision to live, what share of changes now skip a release, how fast you can undo one, and whether anything broke. If the first two get better and the last two don't get worse, it's paying off. Everything else is nice to have, and some of it is misleading.

This is for the person who said yes to moving screens out of the release and now has to say, three months later, whether it was worth it. We'll use no benchmarks, because we don't have trustworthy ones. The only numbers that count are your own.

Start with a baseline you wrote down

You can't show an improvement without a "before". If you priced the wait or picked a first screen, you already have it: the last few changes, with the date each was asked for and the date it went live. If you don't, write it today for the last five changes. It takes half an hour and it is the most useful page in the whole project.

The four measures

MeasureWhat it tells youWhere to get itWarning sign
Time from decision to live, typical and worstWhether the wait actually shrankYour ticket tool: date asked, date liveThe typical time fell but the worst case didn't
Share of changes that skipped a releaseWhether the new path is really usedCount last quarter's screen changes and how many needed a buildMost changes still go through the release
Time to undo a changeWhether you can recover quicklyA rehearsal, timedNobody has tried it, or it needs a developer
Things that went wrongWhether speed cost you qualityIncidents and bad screens that reached users, and how long until someone noticedMore problems, or slower to notice them

Swipe sideways to see every column.

The first two measure the benefit. The last two measure the price. You need both, because a faster path that quietly breaks things isn't an improvement, it's a transfer.

What not to count

  • How many screens you've moved. That's activity. A screen nobody changes is not a win.
  • How many times people published. More publishes can mean more value, or more churn. The count alone doesn't say which.
  • A lift you can't separate from everything else. If conversion went up the month you moved a screen, so did the season, the campaign and the pricing change. Without a comparison group the number is a story, not a result. Reading experiment results and what to measure in a paywall test cover how to do it properly.
  • "It feels faster." Feelings are fine as a starting hypothesis and weak as evidence.

Read it honestly

A few habits keep the review from becoming a sales pitch for your own project:

  • Write the target before you look. "Typical wait under two days, no rise in problems." If you decide after, you'll find a way to call it a success.
  • Compare like with like. A wording change today against a new feature last year isn't a comparison. Compare content changes with content changes.
  • Report the worst case next to the typical one. The change that waited three weeks is the one your stakeholders remember.
  • Give it a full quarter. One month mostly measures the learning curve.

Signs it isn't paying off

  • The queue moved instead of shrinking. Changes no longer wait for a release, but they wait for a review, a legal sign-off or "the one person who can publish". That's a process problem, not a tool problem, and governing the changes is the place to fix it.
  • Developers still get tickets for wording. The new path exists but nobody trusts or owns it. Who owns it usually explains why.
  • Very few changes per month. If a screen changes twice a year, you built a faster road to a place nobody visits. That's a reasonable moment to stop, and when not to use it lists the other cases.
  • More incidents, slower to notice. Speed without a safety net is a bad trade. Look at preview, review and rollback before you add more screens.

A twenty-minute monthly review

Once a month, with the same three people, answer three questions: what changed this month and how long did each take, did anything go wrong and how fast did we notice, and what would we move next or stop moving. Keep a short note. After a quarter you'll have a record that's better than any argument.

When this is more than you need

If you moved one screen as a trial, you don't need a dashboard. Compare that screen's last three changes with the next three and decide. The four measures start to matter when several teams are changing screens and nobody has the whole picture.

Sources

Go further