ROI Factory Tools

ROI Factory Mid-Size Data™ Testing Planner

How much can your traffic tell you, and which marketing test should run next? Enter your own numbers or keep our starting assumptions. Every number here can be changed. Not enough traffic? Plan a paid test.

Projections from assumptions. Nothing here reads your live data. The starting numbers are examples, not industry benchmarks: replace them with yours.

1. Your channels

Each channel is a lane you can test in: a page, an ad account, a social page, an email list. Enter how many people it reaches in a month and how many of them do the thing you care about.

Channel

Name

Reach a month

Result rate

Results a month

Smallest lift you can see in
2 / 4 / 8 weeks

Good for

2. Your assumptions

These settings apply to every calculation below. The defaults follow the white paper's method.

Confidence How sure you want to be that a win is real. Easy-to-undo changes can use 80 to 90%; pricing or brand changes need 95% or more. Power How often the test catches a real lift of the size you're looking for. 80% is the usual setting. Versions in each test Every extra version splits your traffic further and makes each test slower. Longest test you'll run (weeks) Ideas that need longer go to Watch. Kohavi suggests questioning tests that won't finish in a typical 2 to 6 week window. Typical chance an idea works (%) Most tested ideas don't win: published success rates run from about 8% to about 33%. We start at 20 to 30% and raise it only for evidence. Value of one result ($, optional) What a lead or sale is worth to you. Used only for the dollar projections in section 4.

3. Your testing lane

Pick a channel to see how its detection limit falls as a test runs longer.

Channel

Test length

Reach in test

Results in test

Smallest lift you can see

To catch a lift of % this channel needs –.

4. Your test ideas

List what you're thinking of changing. The planner sorts each idea into Decide (just do it), Research (learn more first), Test or Watch (too small to see yet), then ranks the tests by value per week of your lane.

Channel

What kind

Ladder rung

Hard to undo?

Lift if it works

Chance it works

Build weeks

Result

Your queue

Each channel is its own lane, so a website test and an ad test can run at the same time. Ideas are ranked within each channel.

Score = (results a month through the change × lift if it works × chance it works) ÷ lane-weeks (weeks to detect the lift, plus build weeks). Projections assume the lift and chance you entered; real results vary.

5. Buy traffic for a test

Not enough traffic of your own? Buy it. Pick what you want to improve, give us a starting budget and time, and the planner says what that test costs, which channel fits it, and whether your budget and time are enough.

What do you want to improve? Channel Budget for the test ($) Weeks you can run it Smallest lift worth finding (%) Bigger, bolder changes need a smaller test.

Not on the list: "lower cost per sale". Too many things move it at once to prove with a test in a few weeks at a sensible cost. Test the steps that lead to it instead.

The same test in each channel

Channel

Starting rate

Paid for

Needed for the test

Cost

Weeks at your spend

Smallest lift your budget can see

Cost and response assumptions

Starting numbers are 2025 all-industry averages where published [6][7][8], and our own assumptions where not. Yours will differ by industry: replace them with your own numbers.

Channel

Cost per click ($)

Click rate (%)

Conversion rate on clicks (%)

Engaged-visit rate (%)

Source

6. The decision rule for your next test

Write this down before the test starts. It stops moved goalposts, peeking and arguments afterward. This is the rule for the next test in the channel picked in section 3.

7. Automate the next step

Once the rule is written, software can watch the test and act when a threshold is reached. A person still approves the queue and anything hard to undo.

When this happens	Do this automatically	Person needed?
The traffic split looks wrong	Pause the test and alert	Yes, to find out why
A guardrail measure gets clearly worse	Stop the new version and send everyone to the original	Review afterward
The new version passes the win bar	Send all traffic to it, for easy-to-undo changes like ads and creative	Yes, for anything hard to undo
A meaningful difference is no longer likely	End the test, log "no meaningful difference" and start the next test in the queue	No
The lift is far bigger than past results	Hold and alert: surprising wins are usually errors	Yes
The longest run time is reached	End, log the estimate and its range, re-score the queue	No

When this happens

Do this automatically

Person needed?

The traffic split looks wrong

Pause the test and alert

Yes, to find out why

A guardrail measure gets clearly worse

Stop the new version and send everyone to the original

Review afterward

The new version passes the win bar

Send all traffic to it, for easy-to-undo changes like ads and creative

Yes, for anything hard to undo

A meaningful difference is no longer likely

End the test, log "no meaningful difference" and start the next test in the queue

The lift is far bigger than past results

Hold and alert: surprising wins are usually errors

Yes

The longest run time is reached

End, log the estimate and its range, re-score the queue

With your real numbers, this runs itself.

For ROI Factory clients, the planner opens from ROI Factory Mission Control™ with your channels already filled in from Google Analytics, Search Console, your ad accounts and your social pages, and they stay current. A domain Expert in the Loop™ checks everything we deliver.

Free Competitive Marketing Analysis

How the planner works

The planner uses the standard sample-size formula for comparing two rates. The number of people needed in each version is about (z₁ + z₂)² × 2 × p × (1 − p) ÷ (p × lift)², where p is your current rate, z₁ comes from the confidence level and z₂ from the power. With 95% confidence and 80% power this is Evan Miller's well-known rule of thumb of about 16 × variance ÷ effect² [1]. More than two versions split the traffic and tighten the confidence for each comparison.

Because p × n is roughly the number of results in each version, what you can detect depends mainly on how many leads, sales or clicks you get, not on raw traffic. When a test that's too small does show a win, the measured lift is usually exaggerated [2].

The confidence guidance follows Georgiev's review of 1,001 tests, where practitioners chose thresholds between 80% and 95% to match the stakes [3], and Google Ads, whose experiment reports default to an 80% confidence interval [4].

For a paid test, the planner turns the same formula into money: the people needed in each version × the number of versions × what one of them costs (cost per click for page and offer tests; cost per click × click rate for ad tests, which count times shown). Ad tests are easy to undo, so they start at 90% confidence; page and offer tests at 95%.

The Decide, Research, Test and Watch sort, the testing ladder, the scoring formula and the "Good for" bands are ROI Factory's own method, not an industry standard. Social page comparisons (one style of post against another) are directional, because posts aren't randomly split the way a website test is.

Sources

Evan Miller, "How Not To Run an A/B Test" (2010)
Kameleoon, "Ronny Kohavi shares how to accelerate innovation by getting results you can trust" (2022)
Georgi Georgiev, "What Can Be Learned From 1,001 A/B Tests?", Analytics-Toolkit.com (2022)
Google Ads Help, "Monitor your experiments"
Kohavi, Deng and Vermeer, "A/B Testing Intuition Busters", KDD 2022 (success rates of tested ideas)
Search Engine Land, "Google Ads costs keep rising, but conversion rates improved in 2025" (LocaliQ data: search click rate 6.66%, cost per click $5.26, conversion rate 7.52%)
Search Engine Land, "Facebook ad costs jump 21% in 2025, but still beat Google" (traffic campaigns: click rate 1.71%, cost per click $0.70)
WebFX, "LinkedIn benchmarks" (sponsored content click rate about 0.4 to 0.65%, cost per click about $5 to $10)