Experiments & A/B Testing

A/B Test Significance Calculator

Enter visitors and conversions for your control and your variant. Get both conversion rates, the lift, a p-value and confidence interval, a check that your traffic split is healthy, and how many visitors you would need if the result is not conclusive yet.

Your test

Enter visitors and conversions for both variants and press Calculate, or load the example.

What this calculator does

An A/B test shows two versions of a page, email or feature to two random groups of users and compares how often each group converts. The question is always the same: is the difference real, or could random variation explain it? This calculator answers it with a two-proportion z-test, the standard test for comparing two conversion rates.

Beyond the p-value, it reports the absolute difference in percentage points with a confidence interval, the relative lift, a confidence interval for each variant’s rate, a sample ratio mismatch check, and the number of visitors you would need to detect the observed difference reliably.

How the test works

Each variant’s conversion rate is p = conversions / visitors. Under the null hypothesis both versions convert at the same rate, estimated by pooling the two groups:

p̂ = (xA + xB) / (nA + nB)
z = (pB − pA) / √(p̂(1 − p̂)(1/nA + 1/nB))

The p-value comes from the standard normal distribution. The confidence interval for the difference uses each group’s own rate instead of the pooled one: (pB − pA) ± z* √(pA(1 − pA)/nA + pB(1 − pB)/nB). Each variant’s own interval is a Wilson score interval, which stays accurate when conversion rates are low. The z-test is equivalent to a chi-square test on the 2 × 2 table without continuity correction: χ² = z².

Worked example: a new checkout page

Press Load example. An Indian e-commerce site tested a simplified checkout for two weeks. The old checkout (A) had 18,240 visitors and 912 orders; the new one (B) had 18,105 visitors and 1,031 orders.

  • Conversion rates: 5.00% for A and 5.69% for B, a difference of 0.69 percentage points.
  • Relative lift: 13.9%. For every 100 orders the old checkout produced, the new one produces about 114.
  • Significance: z = 2.94, p = 0.0032. Significant at the 95% level.
  • 95% confidence interval for the difference: 0.23 to 1.16 percentage points, so the true lift could be anywhere from about 5% to 23%.
  • Traffic split: 18,240 vs 18,105 is well within chance for a 50/50 split (p = 0.48), so there is no sample ratio mismatch.

Had the test stopped at half the traffic with the same rates, it would have shown p = 0.039: still significant, but only just, and with an interval reaching almost down to zero. The calculator estimates about 16,500 visitors per variant are needed to detect a lift of this size with 80% power, which this test reached.

Statistical vs practical significance

With enough traffic, even a 0.05 percentage point lift becomes significant. Before shipping a winner, convert the lower end of the confidence interval into money: if B’s worst plausible lift does not cover the cost of building and maintaining it, a significant result is not enough on its own. The reverse also holds: a non-significant test with a wide interval has not shown there is no effect, only that this test was too small to tell.

Getting the same result in Python

from statsmodels.stats.proportion import proportions_ztest, confint_proportions_2indep

z, p = proportions_ztest(count=[1031, 912], nobs=[18105, 18240])   # B first, then A
print(z, p)                                                         # 2.943, 0.00325

low, high = confint_proportions_2indep(1031, 18105, 912, 18240, method='wald', compare='diff')
print(low, high)                                                    # 0.00232, 0.01157

In Excel, compute both rates, the pooled rate and z with the formulas above, then =2*(1-NORM.S.DIST(ABS(z),TRUE)) for the two-tailed p-value.

Common mistakes

  • Peeking. Checking daily and stopping at the first significant result inflates false positives far beyond 5%. Fix the sample size first.
  • Testing many variants or metrics at once and declaring a winner on whichever one crossed p < 0.05. With 20 comparisons, one will usually pass by chance.
  • Ignoring a sample ratio mismatch. It almost always signals a bug that also biases the conversion numbers.
  • Stopping before a full weekly cycle. Weekday and weekend shoppers behave differently, and new designs often get a short-lived novelty bump.

Related learning on Codeayan

FAQ

A/B testing questions

What does "statistically significant" mean for an A/B test?

It means the difference in conversion rate between A and B is larger than random variation would usually produce. At a 95% confidence level, if the two versions truly performed the same, a difference this large would show up less than 5% of the time. It does not tell you the lift is large enough to matter for the business.

Can I check the result every day and stop when it becomes significant?

No. Checking repeatedly and stopping at the first significant result (called peeking) sharply raises the false positive rate, because random swings will cross the threshold at some point. Decide the sample size before the test starts and evaluate once it is reached.

What is a sample ratio mismatch?

If you planned a 50/50 split but one variant received noticeably more visitors than chance allows, something in the setup is broken: a redirect that drops users, a bot filter applied to one arm, or a tracking bug. When the calculator flags a mismatch, fix the cause before trusting the conversion result.

Should I use a one-tailed or two-tailed test?

Use two-tailed unless you decided in advance that you will only act if B is better and would treat a worse B exactly like no change. A two-tailed test also warns you when the new version is hurting conversions, which is often the more important finding.

Why is the result not significant even though B has a higher conversion rate?

The test does not yet have enough visitors to separate the difference from noise. The calculator shows roughly how many visitors per variant you would need to detect a lift of the size you observed with 80% power.

More tools

Keep going