What this calculator does
An A/B test shows two versions of a page, email or feature to two random groups of users and compares how often each group converts. The question is always the same: is the difference real, or could random variation explain it? This calculator answers it with a two-proportion z-test, the standard test for comparing two conversion rates.
Beyond the p-value, it reports the absolute difference in percentage points with a confidence interval, the relative lift, a confidence interval for each variant’s rate, a sample ratio mismatch check, and the number of visitors you would need to detect the observed difference reliably.
How the test works
Each variant’s conversion rate is p = conversions / visitors. Under the null hypothesis both versions convert at the same rate, estimated by pooling the two groups:
z = (pB − pA) / √(p̂(1 − p̂)(1/nA + 1/nB))
The p-value comes from the standard normal distribution. The confidence interval for the difference uses each group’s own rate instead of the pooled one: (pB − pA) ± z* √(pA(1 − pA)/nA + pB(1 − pB)/nB). Each variant’s own interval is a Wilson score interval, which stays accurate when conversion rates are low. The z-test is equivalent to a chi-square test on the 2 × 2 table without continuity correction: χ² = z².
Worked example: a new checkout page
Press Load example. An Indian e-commerce site tested a simplified checkout for two weeks. The old checkout (A) had 18,240 visitors and 912 orders; the new one (B) had 18,105 visitors and 1,031 orders.
- Conversion rates: 5.00% for A and 5.69% for B, a difference of 0.69 percentage points.
- Relative lift: 13.9%. For every 100 orders the old checkout produced, the new one produces about 114.
- Significance: z = 2.94, p = 0.0032. Significant at the 95% level.
- 95% confidence interval for the difference: 0.23 to 1.16 percentage points, so the true lift could be anywhere from about 5% to 23%.
- Traffic split: 18,240 vs 18,105 is well within chance for a 50/50 split (p = 0.48), so there is no sample ratio mismatch.
Had the test stopped at half the traffic with the same rates, it would have shown p = 0.039: still significant, but only just, and with an interval reaching almost down to zero. The calculator estimates about 16,500 visitors per variant are needed to detect a lift of this size with 80% power, which this test reached.
Statistical vs practical significance
With enough traffic, even a 0.05 percentage point lift becomes significant. Before shipping a winner, convert the lower end of the confidence interval into money: if B’s worst plausible lift does not cover the cost of building and maintaining it, a significant result is not enough on its own. The reverse also holds: a non-significant test with a wide interval has not shown there is no effect, only that this test was too small to tell.
Getting the same result in Python
from statsmodels.stats.proportion import proportions_ztest, confint_proportions_2indep
z, p = proportions_ztest(count=[1031, 912], nobs=[18105, 18240]) # B first, then A
print(z, p) # 2.943, 0.00325
low, high = confint_proportions_2indep(1031, 18105, 912, 18240, method='wald', compare='diff')
print(low, high) # 0.00232, 0.01157
In Excel, compute both rates, the pooled rate and z with the formulas above, then
=2*(1-NORM.S.DIST(ABS(z),TRUE)) for the two-tailed p-value.
Common mistakes
- Peeking. Checking daily and stopping at the first significant result inflates false positives far beyond 5%. Fix the sample size first.
- Testing many variants or metrics at once and declaring a winner on whichever one crossed p < 0.05. With 20 comparisons, one will usually pass by chance.
- Ignoring a sample ratio mismatch. It almost always signals a bug that also biases the conversion numbers.
- Stopping before a full weekly cycle. Weekday and weekend shoppers behave differently, and new designs often get a short-lived novelty bump.