A/B testingCRO agencyrevenue per visitor

Testing velocity: why tests per month is the wrong question to ask a CRO agency

Testing velocity counts how many A/B tests a team ships, not whether they make you money. Why buying CRO by test count sets up the wrong conversation, and what to ask a Shopify CRO agency for instead.
Paddy McLarnon
Written by
PM Digital | Founder Paddy McLarnon
Shopify CRO & performance agency for CPG and subscription brands
6 min read
Natrava cart upgrade block from a winning subscription test

When a founder asks a CRO agency how many tests they will get each month, they are asking about testing velocity, but they usually mean something else: will this make our paid traffic more profitable, and how will we know? Velocity is a fair thing to want, but it is a poor way to choose who to work with.

Most CRO proposals open with output. You will see a line like "four to six tests and two to three landing pages a month", and a price underneath. It looks clear, and it is easy to compare against the next agency's quote. The trouble is that it tells you what will be made, not what will change in your business, and it quietly sets up the wrong conversation for the months that follow.

The term

What is testing velocity?

Testing velocity is the number of experiments a team launches or completes in a given period, usually counted per month. It is a useful health check for an in-house programme, because a team that rarely ships tests rarely learns anything. It becomes a problem when it is the thing you buy from an agency, because it measures activity rather than whether the tests were worth running.

The real question

What you are actually buying from a CRO agency

A founder rarely wants six A/B tests for their own sake. They want to know why acquisition costs keep rising, why fewer first orders turn into subscriptions than they should, or why more traffic is not turning into more profitable revenue. Tests are how you find and prove the answers, so they matter, but they are the method rather than the goal.

That is why we judge every test on revenue per visitor and, for subscription brands, on subscription take rate. Conversion rate on its own can rise while order value or subscriber share falls, and a programme counted by tests shipped will happily call that a win. If you want the detail on that metric, our guide to revenue per visitor walks through it, and if you are new to working with one, start with what a CRO agency does.

Natrava cart upgrade block offering subscribe and save over 20% with a free frother, free shipping
Natrava: the cart upgrade block from the winning test. It lifted subscription revenue per visitor by 9%, which is the number we judged it on. Read the Natrava case study.

Why counting fails

Three problems with buying CRO by testing velocity

The first problem is that volume becomes the scoreboard. If the contract says six tests and the team ships four, the conversation that month is about the missing two, even if one of the four moved revenue per visitor in a way that pays for the whole year. The team starts optimising for things that are quick to launch rather than things that are worth knowing.

The second is that a test count hides the size of each test. Changing a button label and rebuilding a subscription offer are both one test, but one takes an afternoon and the other needs research, new copy, design, a careful build and QA across your subscription app. On Cadence we tested ten different offer structures before the subscription take rate settled at around 70%, and each of those was a substantial piece of work.

The third is traffic. Testing velocity is capped by how many visitors you have, because a store can only run as many tests as its traffic can read in a sensible time, and a promise of six a month either assumes traffic you may not have or pushes the team into calling results early.

Kiko Vitals buy box before and after: the after version leads with subscribe and save, shows free
Kiko Vitals: the buy box before and after we rebuilt it. Against the four months before, subscription sales were up 64%, store-wide, so the figure includes Kiko's own marketing too. Read the Kiko Vitals case study.

A better way to judge it

What to ask a CRO agency for instead

You still need to know what you will get each month, so the answer is not to drop output altogether. It is to separate three things that usually get blurred into one line on a proposal.

Ownership
Who is responsible for priorities, the roadmap and reporting, and who you speak to when something needs deciding.
Execution
The research, copy, design, Shopify development, QA and testing that turn a hypothesis into something live on your store.
Commercial improvement
The numbers the work is judged on, such as revenue per visitor and subscription take rate, and how often you see them.

A good proposal then gives you an illustrative month rather than a fixed quota. One month might be two offer tests and a new landing page, another might be one large subscription rebuild and the rollout of last month's winner. You should be able to see roughly how much gets built, and the agency should be able to explain why this month looks the way it does.

MyoMaster winning cart design with clearer product cards, delivery timing for each product and the
MyoMaster: one cart test, with revenue per visitor up 45%. One test, but a lot of research and build behind it. Read the MyoMaster case study.

In practice

How we structure it at PM Digital

On the CRO & Scale Plan your pod is a CRO strategist, copywriter, designer, Shopify developer, QA specialist and project manager. We agree the monthly scope with you up front, based on your traffic and where the biggest opportunities are, and every test runs in AB Genius, our own Shopify testing app, or in Intelligems if your brand already uses it.

Because we are judged on results rather than volume, we also show the record honestly. With Bond Touch, 8 of 13 completed tests won, and the ones that lost still taught us something we used in the next round. If you want to see how a programme adds up over a year, the Cadence case study shows store revenue up 389% in 12 months on our CRO & Scale Plan, with the caveat that the figure is store-wide and includes Cadence's own acquisition, brand and retail growth.

If you are earlier on, or want to try us on one problem first, the CRO sprint is a different kind of project rather than a smaller plan, and we explain the difference in Shopify CRO services: a one-off sprint or an ongoing programme. For what agencies charge and what each fee usually buys, see Shopify CRO agency cost in the UK.

Questions

Speed only helps if each test is sound, so read how long to run an A/B test and how to brief a CRO test before you add more tests to the roadmap.

Frequently asked questions

What is a good testing velocity for a Shopify brand?
As many tests as your traffic can read properly and your biggest opportunities justify. A store with modest traffic might only support a couple of meaningful tests at once, so a fixed promise of six can push a team to call results early.
Should a CRO proposal include a number of tests?
It should show you roughly what gets built each month, ideally as an example month, and say who owns priorities and which metrics the work is judged on. A hard quota on its own tends to turn the relationship into a debate about volume.
What should a CRO programme be judged on?
Revenue per visitor, and for subscription brands the subscription take rate as well. Conversion rate alone can rise while order value or subscriber share falls.
Talk with Paddy
Want a plan built around your numbers, not a test count?
Book a call with Paddy, our founder. It is a chat to see where the brand is and what it needs, and we will only audit if it is needed.
Talk with Paddy
A chat with our founder about where the brand is.
We'll look at what you need, audit if it's needed, then talk about how we could work together. No deck, no obligation.
Book a discovery call WhatsApp us
Paddy McLarnon
Written by
Paddy McLarnon
Founder of PM Digital. Runs full-funnel CRO programmes for CPG and subscription brands on Shopify, and writes about offers, positioning and what the tests actually show.

How we work

Built for what happens after the click.

Most brands don't need more traffic. They need the traffic they already pay for to convert.

We work on what happens after the click, on the CRO and Scale Plan. Decisions are driven by the data in the CRO Brain, with PMD Cortex AI supporting the team, and a programme of tests rather than one lucky win.

The founder and the team

Paddy McLarnon, founder, and the team that does the work.

Paddy McLarnon founded PM Digital and has been a name in CRO for the last four years. He takes the first call with every brand himself, and stays close to the strategy. He has spoken at DTC Growth Summit in Miami and New York, and hosts the Funnel First podcast.

Meet the agency
Paddy McLarnon, founder of PM Digital, speaking at DTC Growth Summit Miami
Founder, speaking at DTC Growth Summit Miami

Our team

The in-house team behind the work.

Your pod does the work, and the whole team backs it. Plus you, in a dedicated Slack channel with all of them.