- The floor is orders, not visits. A 20% lift takes about 420 per version, 10% about 1,600, and 5% about 6,300, at almost any conversion rate.
- Kohavi’s 200,000 means users in one experiment, at 5% conversion and a 5% lift. [1] At 2% conversion it’s about 630,000.
- Real effects are small. Qubit found 90% of tests moved revenue by under 1.2%. [8] One vendor’s median lift across 1,001 tests is 0.08%. [9]
- In a four-week test, a store with 500 orders a month can’t record a win under about 18%. Wins are inflated by construction.
- “Mostly noise” overstates it: measured false discovery on Optimizely was 18-25%, on clicks. [6] The size of the win is the part you can’t trust.
- If the change is built and reversible, picking the higher arm can beat waiting for significance. [7] Just don’t report the lift.
Someone proposes testing a new product page. The tool is live, the design is done, and the plan is to run it for a month. Nobody has asked the one question that decides whether this is an experiment or a coin toss with a dashboard: how many orders will be in the test?
Those are exact calculations at 95% confidence and 80% power with a 2% baseline, and they barely move with your conversion rate. Going to a 5% baseline changes the 10% row from 1,613 orders to 1,561. So we prefer to talk about orders. Traffic is one step removed, and it hides the fact that dashboards count sessions while the formula wants users.
This is also where the most repeated number in the field goes wrong. Kohavi’s “200,000” circulates as a monthly traffic bar. His wording is “about 200,000 users in an experiment,” at a 5% conversion rate and a 5% detectable lift. [1] Run his inputs at 2%, where many stores sit, and it’s about 630,000 users, or roughly 685,000 visitors a month for a four-week test. He also tells smaller sites to start, looking for big effects. [4] The floor moved a hundredfold because the question moved.
Here’s the calculator. It runs backward: tell it the lift you’d act on, and it tells you how long you’d wait.
How long your test has to run
Count visitors who actually reach the page you’re changing, rather than whole-site traffic.
Model: the familiar rule of 16 × rate × (1 - rate) / difference², per version, using the midpoint of the two rates. [11] It assumes a fixed length, an even split, and no multiple-comparison correction. Over six weeks it gets worse in practice, as people return and script-set cookies expire: Safari deletes them after 7 days without interaction. [18]
What real tests produce
Knowing what you can detect only helps if you know what’s out there to detect, and the answer is unkind. Qubit, a vendor, analyzed over 6,700 large experiments and found 90% moved revenue by less than 1.2%, up or down. [8] Analytics Toolkit’s 1,001 tests have a median lift estimate of 0.08%. [9] At Bing, winners improved key metrics by 0.1% to 1%. [17] Airbnb’s search team ran 250 experiments, and the 20 winners averaged 0.3%. [1] Kohavi’s own framing is that a 5% lift on 5% conversion “is considered a large win.” [3]
Those are big, mature sites, and nobody has published the same distribution for small ones. Smaller, less optimized sites plausibly have more upside. Even so, set those effects against what a store can see in a month:
| Orders a month | Visitors at 2% | Smallest lift you can reliably detect | Smallest lift that can ever show as a win |
|---|---|---|---|
| 100 | 5,000 | 66.5% | 40.4% |
| 250 | 12,500 | 40.0% | 25.6% |
| 500 | 25,000 | 27.5% | 18.1% |
| 1,000 | 50,000 | 19.1% | 12.8% |
| 2,000 | 100,000 | 13.3% | 9.0% |
| 5,000 | 250,000 | 8.3% | 5.7% |
| 10,000 | 500,000 | 5.9% | 4.0% |
| 20,000 | 1,000,000 | 4.1% | 2.9% |
That table comes from our exact calculation, for a four-week test with two versions. Look at the right-hand column. It’s where the trouble lives.
A win has a minimum size
To clear significance, a result has to be about 1.96 standard errors from zero. The lift you can detect with 80% power sits at 2.8 standard errors. So the smallest possible win is about 0.7 of your detectable lift. In a four-week test, a store with 500 orders a month can’t record a win under about 18%. If real effects run under 5%, then every win it ever logs is at least 3.6 times too big. That isn’t bad luck. It’s the filter.
Gelman and Carlin named this a Type M error: low power doesn’t just miss effects, it exaggerates the ones it catches. [5] Kohavi and colleagues put it flatly: an underpowered significant result “must exaggerate the true treatment effect.” [3] We simulated it with two binomial arms, 200,000 runs per row, at a 2% baseline.
| Visitors a month | True lift | How often it wins | Average reported win | Exaggeration |
|---|---|---|---|---|
| 10,000 | 5% | 5.3% | 41.6% | 8.3× |
| 25,000 | 5% | 7.7% | 24.7% | 4.9× |
| 50,000 | 5% | 11.5% | 17.2% | 3.4× |
| 100,000 | 5% | 18.7% | 12.2% | 2.4× |
| 500,000 | 5% | 66.9% | 6.2% | 1.2× |
You can see the direction in the wild, though it isn’t a clean measure of exaggeration. Analytics Toolkit’s winners have a mean lift of 15.9%, against 2.08% for all tests. [9] Kohavi’s paper dissects a published 337% win on 82 and 75 visitors, a test with 3% power to find a 10% change. [2] And one claimed 55% lift, from 919 visits, replicated three times at 0.16%, 0.29%, and 0.73%. [3]
How wrong is “mostly noise”?
We started out assuming low-traffic winners were mostly false. It’s more complicated than that, and you should know why. Berman and Van den Bulte measured false discovery on 2,766 Optimizely experiments: 18-25% at 5% significance. They attribute it mainly to a high share of true nulls, about 70%, rather than to low power. [6] Their metric, though, is clicking anywhere on the page, at a rate near 68%, where small tests are well powered. Nobody has published the same analysis on purchases.
Our simulation shows why this can’t be pinned down: at 25,000 visitors, the share of wins that are false runs from 4% to 44% depending on how you assume true effects are spread. The average win overstates the truth in every scenario, from 1.6 times to well over 100. So don’t tell your team the winners are fake. Tell them the number is. Direction is often right. Size almost never is.
That’s also why the usual remedies disappoint.
| The advice | What it buys | What it costs |
|---|---|---|
| Lower significance to 80% [13] | About a quarter off your detectable lift | False wins rise from 42% to 57% at a 15% success rate, in our calculation, and Optimizely’s own docs warn that a win from an underpowered test is unlikely to hold up [14] |
| Test bigger changes | Easier to see, if they move anything | Qubit’s page redesigns averaged +0.2%, and Kohavi says most big jumps fail [8][4] |
| Use a micro-conversion [13] | Much more volume | Correlation with sales is 0.43 to 0.61, from vendor data [15] |
| Go sequential [12] | Saves time when a variant is a big winner | Almost nothing under a small true effect, in our run of Miller’s method |
| Stack a one-sided test at 90%, sequential stopping, and an assumed 20% variance cut | About one tier of sensitivity | A 25,000-visitor store goes from about 27% to about 16%, which still isn’t 5% |
But a test that can’t reach significance can still pay
Here’s the part that makes this more than a shrug. Significance answers “am I sure?” A store owner’s real question is “which version should I keep?” Feit and Berman showed that the profit-maximizing test is far smaller than a hypothesis test, and that tests “that will never reach statistical significance can still have substantial benefit,” assuming no deployment cost. [7] Georgiev’s simulations go further: any testing beat none, even at 5,000 users, and shipping everything untested had a median effect of -11.5%. [10] That rests on his assumed wide spread of effects, and he adds that it probably won’t pay small firms once fixed costs are in.
The arithmetic for a built, reversible change is striking. At 25,000 visitors, assuming true lifts spread around 5%, simply keeping the higher arm captures 48% of the available gain. Requiring p below .05 captures 11%, because it almost never ships anything. The price is keeping the worse version about a third of the time, usually by a small amount, and you can’t honestly report the lift.
If you’re going to record a number, discount it first.
How much of a win to believe
Use the conversions in your control arm. For the spread, 1-3% suits a well-optimized site and 5-10% a site that has done little testing. Those ranges are our judgment.
Model: observed lift × spread² / (spread² + 20,000 / conversions). It assumes true effects are centered on zero, so a genuine outlier gets shrunk too far. Our simulation found the estimate within about 4% of the true average among wins.
What to do at your size
These tiers are our synthesis, keyed on conversions a month on the page under test. If you’re lead-gen, count qualified leads or sales. A form fill is itself a proxy, so a site with 1,250 leads a month and a 10% close rate can only read about 59% on sales.
| Orders a month | Can detect | What to do |
|---|---|---|
| Under 170 | Above 50% | Don’t split test on orders. Fix what’s broken and ship it. Run five-user usability rounds, which find problems but don’t size them. [16] Copy patterns with a track record. Don’t buy a tool or retainer sold on win rates |
| 170-900 | 20-50% | One test at a time, on the proposition: offer, price, guarantee, funnel steps. Fixed length of four to eight weeks. Keep the higher arm if it’s reversible, and write down the discounted lift. Read add-to-cart, with orders as the guardrail |
| 900-3,500 | 10-20% | About a test a month on large changes. Validate your proxies against your own orders. Rerun big wins before you build them into a forecast |
| 3,500-14,000 | 5-10% | A real program, with the detectable lift fixed in advance. Effects under 5% are still inflated, so replicate surprises |
| Above 14,000 | Under 5% | Kohavi’s zone at a 2% rate. Test anything risky |
Before any test, ask three things. Is the lift you’d act on above what you can detect? Is the true lift plausibly that big? And is the decision worth more than the test costs? For a store with $480,000 in annual revenue, one decision is worth somewhere from about $200 to $14,000 by our estimate, depending on the spread. Hold that next to the retainer.
The same arithmetic for ad creative is in our piece on creative testing, and our earlier look at AI and CRO tells you to measure every lift with a controlled test. For much of its audience, that advice runs straight into the table above. The honest version is to measure what you can, know your floor, and stop putting the lift in the deck.
Sources
- Ron Kohavi · Why 5% should be the upper bound of your MDE in A/B teststhe 200,000 wording and its inputs, November 2023
- Kohavi, Deng, and Vermeer · A/B testing intuition bustersKDD 2022, the 16 rule, success rates, and the 337% dissection
- Kohavi and colleagues · Power analysis is essentialpreprint of the Econ Journal Watch article, 2026, including co-authors Gelman and Imbens
- Kohavi, Tang, and Xu · Trustworthy Online Controlled Experiments, chapter 12020, on when to start and on big jumps failing
- Gelman and Carlin · Beyond power calculations: assessing Type S and Type M errorsPerspectives on Psychological Science 2014, Type S and Type M errors
- Berman and Van den Bulte · False discovery in A/B testingManagement Science 2021, 2,766 Optimizely experiments, metric is clicks
- Feit and Berman · Test & roll: profit-maximizing A/B testsMarketing Science 2019, profit-maximizing test sizes, assuming no deployment cost
- Browne and Jones · What works in e-commerce: a meta-analysis of 6700 online experimentsQubit 2017, over 6,700 large experiments, vendor data with a PwC-assured method
- Analytics Toolkit · What can be learned from 1,001 A/B testsvendor data from advanced users
- Analytics Toolkit · The business value of A/B testingvendor simulation, October 2024, on an assumed wide spread of effects
- Evan Miller · How not to run an A/B test2010, the rule of thumb and the peeking problem
- Evan Miller · Simple sequential A/B testing2015, and the warning that it runs slower when no effect exists
- Optimizely · Test tips for low-traffic sitesvendor guidance on lowering significance and on micro-conversions
- Optimizely · How long to run an experimentthe vendor’s warning that small-sample improvements may not hold up
- GoodUI · Do adds to cart or progression metrics correlate with sales?vendor analysis with correlations of 0.43 to 0.61
- Nielsen Norman Group · Why you only need to test with 5 users2000, which finds problems and doesn’t size them
- Kohavi and colleagues · Seven rules of thumb for web site experimentersKDD 2014, with winners at Bing of 0.1% to 1%
- WebKit · Tracking prevention in WebKitscript-set storage deleted after 7 days without user interaction
Frequently asked questions
How much traffic do I need to A/B test?
Count orders rather than visits. At 95% confidence and 80% power with two versions and a 2% conversion rate, reading a 20% lift takes about 420 orders per version, a 10% lift about 1,600, and a 5% lift about 6,300. The number of orders barely changes with your conversion rate.
Is 200,000 users really the minimum for A/B testing?
Ron Kohavi’s figure is about 200,000 users in one experiment, at a 5% conversion rate and a 5% detectable lift. It is not visitors per month, and it is not a minimum for testing at all. At a 2% conversion rate, the same inputs need about 630,000 users. He also says A/B tests are useful with at least thousands of active users, preferably tens of thousands, for effects of reasonable size.
My test hit 95% significance on low traffic. Is the win real?
The direction may well be right, but the size almost certainly is not. The smallest possible significant win is about 0.7 times your minimum detectable lift. In our simulation of a store with 25,000 monthly visitors and a true 5% lift, wins averaged 24.7%, or 4.9 times the truth.
Can I lower significance to 80% or 90% because I have little traffic?
You can for reversible changes, as long as you stop quoting the lift. Going to 80% takes about a quarter off your minimum detectable lift. In our calculation, on the same test at a 15% success rate, the share of false wins rises from 42% to 57%, while exaggeration only eases from 2.3 to 1.8 times.
Should I test on add-to-cart if I can’t reach significance on purchases?
Use it as a read, with orders as a guardrail. Published correlations between add-to-cart or checkout metrics and sales run from about 0.43 to 0.61, so a lift on the proxy tells you something but does not transfer one to one. That data is from a vendor.
What should a low-traffic site do instead of A/B testing?
Under roughly 170 orders a month, fix what is broken and ship it, run rounds of five-user usability testing, and copy patterns that have held up elsewhere. Between roughly 170 and 900, test one proposition-level change at a time, ship the higher arm if it is built and reversible, and record a discounted lift rather than the observed one.
Are sequential tests a fix for low traffic?
Only partly. They shorten tests when a variant is a large winner. In our simulation of Evan Miller’s method, with a true 5% lift against a 20% design, the test used nearly its full budget anyway. They do not make small effects visible.
Mikołaj Salecki
Writes about media, tech, and AI business for people who actually run digital. Former agency lead. Skeptic of frameworks that read better than they perform.
