Pocketsflow
Start selling free

// a/b test calculator

Is your A/B test winner real?

Enter visitors and conversions for two versions. See the lift, the confidence that it is not chance, each version's likely range, and how many visitors the test needs.

Start selling freeRun A/B tests

Let your agents run and grow your business. Sign in, click Allow, no API key.

A/B test calculator

Version A (control)
Version B (variant)

A conversion is whatever you are testing for: a sale, a sign-up, a click. Each visitor should have seen only one version.

The verdict

Version B wins, at 98.4% confidence.

Version A converts4%
Version B converts5%
Lift (B vs A)+25%
Confidence98.4%
A, 95% range3.46% – 4.54%
B, 95% range4.4% – 5.6%
Visitors needed per version6,745
Where each version's true conversion rate likely isVersion AVersion B
2.9%3.2%3.6%3.9%4.2%4.6%4.9%5.2%5.6%5.9%6.2%
Where each version's true conversion rate likely is
PointVersion AVersion B
2.89% conversion0% of peak0% of peak
2.95% conversion0% of peak0% of peak
3% conversion0% of peak0% of peak
3.06% conversion0% of peak0% of peak
3.11% conversion1% of peak0% of peak
3.17% conversion1% of peak0% of peak
3.23% conversion2% of peak0% of peak
3.28% conversion3% of peak0% of peak
3.34% conversion6% of peak0% of peak
3.39% conversion9% of peak0% of peak
3.45% conversion14% of peak0% of peak
3.5% conversion20% of peak0% of peak
3.56% conversion28% of peak0% of peak
3.62% conversion38% of peak0% of peak
3.67% conversion49% of peak0% of peak
3.73% conversion62% of peak0% of peak
3.78% conversion74% of peak0% of peak
3.84% conversion84% of peak0% of peak
3.89% conversion93% of peak0% of peak
3.95% conversion98% of peak0% of peak
4.01% conversion100% of peak0% of peak
4.06% conversion98% of peak1% of peak
4.12% conversion92% of peak1% of peak
4.17% conversion82% of peak2% of peak
4.23% conversion71% of peak4% of peak
4.28% conversion59% of peak6% of peak
4.34% conversion47% of peak9% of peak
4.4% conversion36% of peak13% of peak
4.45% conversion27% of peak18% of peak
4.51% conversion19% of peak25% of peak
4.56% conversion13% of peak33% of peak
4.62% conversion8% of peak42% of peak
4.67% conversion5% of peak51% of peak
4.73% conversion3% of peak61% of peak
4.78% conversion2% of peak70% of peak
4.84% conversion1% of peak79% of peak
4.9% conversion1% of peak85% of peak
4.95% conversion0% of peak89% of peak
5.01% conversion0% of peak90% of peak
5.06% conversion0% of peak88% of peak
5.12% conversion0% of peak83% of peak
5.17% conversion0% of peak77% of peak
5.23% conversion0% of peak68% of peak
5.29% conversion0% of peak58% of peak
5.34% conversion0% of peak49% of peak
5.4% conversion0% of peak39% of peak
5.45% conversion0% of peak31% of peak
5.51% conversion0% of peak23% of peak
5.56% conversion0% of peak17% of peak
5.62% conversion0% of peak12% of peak
5.68% conversion0% of peak8% of peak
5.73% conversion0% of peak5% of peak
5.79% conversion0% of peak3% of peak
5.84% conversion0% of peak2% of peak
5.9% conversion0% of peak1% of peak
5.95% conversion0% of peak1% of peak
6.01% conversion0% of peak0% of peak
6.07% conversion0% of peak0% of peak
6.12% conversion0% of peak0% of peak
6.18% conversion0% of peak0% of peak
6.23% conversion0% of peak0% of peak

A two-proportion z-test, two-sided. Significant means under a 5% chance of a gap this size if the versions were the same. Visitors needed is per version, to detect this gap at 95% confidence and 80% power. Decide your sample size first: checking until it looks significant finds winners that are not real.

Two versions, one conversion each visitor. A frequentist test, two-sided.

// the formulas

Signal, or noise.

The worked examples use the calculator's defaults: 5,000 visitors on each version, 200 sales on A and 250 on B.

  • Conversion and lift

    rate = conversions / visitors · lift = B / A − 1

    200 sales from 5,000 visitors is 4%; 250 is 5%. B converts 25% better.

  • Confidence

    z = (B − A) / pooled standard error

    A two-proportion z-test. The example's z of 2.41 means 98.4% confidence: a gap this big would be rare if the versions were the same.

  • Sample size

    visitors per version for 95% confidence, 80% power

    To reliably catch a 4% → 5% change you need about 6,745 visitors a version. Smaller gaps need far more: 4% → 4.4% takes about 39,475.

On Pocketsflow's Pro plan, A/B tests on price, headline, cover and checkout layout run on real visitors and tell you in plain words when one version is clearly ahead. See A/B testing or compare plans.

// faq

Common questions.

How do I know if my A/B test is significant?

Compare the two conversion rates with a two-proportion z-test and look at the confidence. At 95% or more, the gap is unlikely to be chance. 200 vs 250 sales out of 5,000 visitors each is 98.4%: significant.

Why is my test not significant with a big lift?

Because the sample is small. The same 4% vs 5% with 500 visitors each (20 vs 25 sales) is only 55.4% confidence. Small samples swing a lot by chance, so keep the test running until you reach the visitors you planned.

How many visitors does an A/B test need?

It depends on your conversion rate and the gap you want to detect. From 4% to 5% takes about 6,745 visitors per version at 95% confidence and 80% power. Pick the number before you start and stop there.

Can I stop the test as soon as it looks significant?

No. Checking again and again and stopping at the first significant result finds many winners that are not real. Decide the sample size up front, or use a tool that corrects for repeated looks.

Can Pocketsflow run the test for me?

Yes, on the Pro plan. Test up to four prices, headlines, covers or checkout layouts on real visitors. Each visitor keeps one version, every sale is credited to the version they saw, and one click keeps the winner.

// more free tools

Plan the rest of your launch.

Free, no sign-up. When you are ready to sell, the same numbers run your store: see pricing.

Open your store today.

Sell your first product in minutes, keep more of every sale, and let the AI you already use run the busywork.

Start selling freeImport my store

Or start from the AI you already use