Is a CRO agency worth it? Do the arithmetic first
A CRO retainer is priced on test velocity, and tests need conversions rather than traffic. Here is the threshold, worked through with real numbers.
A CRO retainer buys you a testing programme. One test needs roughly 850 conversions to settle a 20% swing, so under that volume you are paying testing prices for opinions.
Is a CRO agency worth it at your traffic level?
A CRO agency earns its retainer when your site produces enough conversions for a test to finish. The working floor is about 850 conversions a month to settle a 20% swing inside four weeks, or about 3,200 a month to settle a 10% one. Below that, buy diagnosis instead.
The usual answer gets framed in revenue or company size. Both pricing guides below scale their tiers that way, by catalogue size, funnel count and technical complexity. Neither mentions the number that decides whether a test can finish.
The number that decides it is conversions. Not visitors, not revenue. Ron Kohavi, who ran experimentation at Microsoft, and Stefan Thomke of Harvard Business School put the entry point plainly: any company with at least a few thousand daily active users can run these tests. A few thousand daily. Most sites reading this do a few thousand monthly.
How many conversions does one A/B test need?
Detecting a 20% relative lift at 95% confidence and 80% power takes about 425 conversions per variant, so roughly 850 across both. A 10% lift takes about 1,600 per variant, roughly 3,200 in total. A 30% lift takes about 390 in total.
- 390 conversions to settle a 30% swing, both variants combined
- 850 conversions to settle a 20% swing
- 3,200 conversions to settle a 10% swing
Those are two-sided tests at the standard thresholds. You can reproduce every one of them in Evan Miller's sample size calculator in about a minute, which is the point of publishing them rather than asserting them.
Why doesn't your conversion rate change that number?
Because a test counts conversions, not visits. Across baseline rates from 0.5% to 5%, the conversions needed to settle a 20% swing move only between 816 and 859. The visitors needed move from 8,158 per variant to 85,862. Same test, same certainty, wildly different traffic.
Your traffic decides how long the test takes. Your conversions decide whether it can finish at all.
This is why a low conversion rate does not disqualify you and a high one does not save you. A 5% converting site with 4,000 visitors has 200 sales a month and cannot settle a 20% question in under four months. Neither can a 1% site with 20,000 visitors. The arithmetic does not care which of those you are.
What does a CRO retainer cost in 2026?
The published 2026 figures disagree with each other. Conversion Rate Store puts typical retainers at $5,000 to $15,000 a month. Convert puts the average at $16,000, with a low end of $2,000. Both sell into this market, in opposite directions, so the floor is the reliable part.
The disagreement is informative. Convert sells testing software to agencies, so a higher published retainer makes its own customers look like viable businesses. Conversion Rate Store sells retainers, so a lower typical figure makes its quote look reasonable next to it. Where they agree is the bottom: $2,000 to $2,500 a month, on a six to twelve month minimum.
Not sure what your gap is worth first? Revslip prices it on your URL.
What does that buy a site doing 112 sales a month?
Take 8,000 visitors a month at a 1.4% conversion rate, with a $90 average order. That is 112 sales. Accumulating the 850 conversions one 20% test needs takes 7.6 months. At a $5,000 retainer, that is roughly $37,900 spent to settle a single question.
- Visitors. 8,000 a month, from analytics.
- Sales. 112 a month at 1.4%.
- One test. 850 conversions, so 7.6 months of traffic for one answer.
- Retainer spent. $15,200 at the $2,000 floor, $37,900 at $5,000.
Now the part that decides it. If that test wins by 20% it adds 22 sales a month, worth $2,016, which repays a $5,000 retainer in two and a half months. Good deal, if the first test wins.
It mostly does not. ConversionTeam audited 2,288 of its own tests and found 19.1% reached statistical significance. Kohavi and Thomke report 10% to 20% of experiments generating positive results at Google and Bing. Call it one in five. Five tests at 7.6 months each is 38 months of traffic before you own one proven winner.
When is a retainer the right call?
Above roughly 850 conversions a month you can settle one 20% question per test cycle, which is what a retainer is priced to deliver. Above 3,200 you can chase 10% swings, and on a site that has already been optimised hard, that is where the remaining money sits.
Worth saying: testing is not everything an agency does. The research, the heuristic review, the checkout walkthrough, that work is real and it does not need statistical power. But you are being billed on test velocity, and that is the part the arithmetic above prices.
Under the threshold, the honest move is to find the gap rather than test into it. Revslip publishes the formula it uses for every estimate:
traffic × CVR gap × AOV × mobile weight = monthly leak
On the same site, 8,000 visitors against a one point conversion gap at $90 gives $7,200 a month on the table. That is a number you can carry into a decision this week rather than in 2029. The longer version of that calculation is in how to calculate conversion loss, and if you want to know whether 1.4% is even the problem, start with is my conversion rate bad.
Below the threshold, run a free audit and get the ranked list first.
Where does this argument break down?
In three places. Revslip sells the alternative, so the threshold flatters us. The 850 figure assumes fixed-horizon testing, and other methods call tests sooner. And a badly broken page can swing far more than 20%, which needs a fraction of the traffic to prove.
The part that undercuts our own pitch Revslip has audited 134 businesses. That is a small sample and a self-selected one, because every one of them arrived already suspecting something was wrong. It biases what we find toward sites that have something to find.
And an agency with a strong hypothesis beats a calculator. The 850 figure assumes you are testing a coin flip. A team that has seen your failure mode on forty other sites is not guessing, which is a real argument for paying one, and the reason a paid audit can be worth it even when a retainer is not.
Common questions
The three that follow a founder home from an agency call: whether the threshold can be dodged, whether a one-off audit carries the same limit, and what to do when the retainer on the table is not a testing retainer at all.
Can I run tests below this threshold?
You can run them. You cannot settle them. A test that never reaches significance still produces a number, and that number is how underpowered programmes generate confident, wrong decisions.
Does a one-off CRO audit have the same problem?
No. An audit is a diagnosis, not an experiment, so it has no sample size requirement. It also carries no statistical guarantee, which is the trade you are making.
What if the agency does not run tests?
Then ask what you are paying monthly for. Research and design work is usually better bought as a defined project, which Convert prices at $2,500 to $4,500 as a one-off conversion audit.
Sample sizes calculated at 95% confidence and 80% power, reproducible in Evan Miller's calculator. Win rates from ConversionTeam's audit of 2,288 tests and from Kohavi and Thomke in Harvard Business Review. Pricing from Conversion Rate Store and Convert, both commercially interested.