What this calculator does
A sample size calculation answers one question before you collect any data: how many observations do you need so that a real effect of a given size is likely to show up as significant? Too few and the test is underpowered, so a genuine improvement looks like noise. Too many and you waste traffic and time that could have gone into the next experiment.
The calculator covers the three cases analysts meet most often. A/B test on conversion rates gives the users per variant, the total, and how many days to run at your traffic. Compare two means handles continuous metrics such as order value or delivery time, with unequal group sizes if you want them. Survey or proportion estimate gives the number of completed responses for a chosen margin of error, with the finite population correction for small populations.
The formulas
Two means: n₁ = (zα + zβ)² σ² (1 + 1/r) / Δ², n₂ = r × n₁
Survey: n₀ = z² p(1 − p) / E², with population N: n = n₀ / (1 + (n₀ − 1) / N)
zα = z1 − α/2 (two-sided) or z1 − α (one-sided), zβ = zpower, Bonferroni: α / (variants − 1)
Each result is rounded up to a whole person. The proportion formula uses the pooled rate under the null hypothesis and the two separate rates under the alternative, which is the standard normal approximation behind most A/B testing tools.
Worked example: UPI one-tap checkout
Press Load example. A Bengaluru D2C brand converts 3.2% of checkout visitors and wants to test a one-tap UPI payment button. A lift smaller than 15% (3.2% to 3.68%) would not justify the engineering work, so that is the minimum detectable effect. The site gets 8,000 checkout visitors a day and half of them go into the test.
- Sample: 22,631 visitors per variant, 45,262 in total, at α = 0.05 (two-sided) and 80% power.
- Duration: 45,262 / 4,000 a day = 11.3, so at least 12 days. Rounding up to two full weeks (14 days) covers each weekday twice.
- Trade-off: the Scenarios tab shows that a 10% MDE would need about 2.2 times as many visitors, which is why the MDE is the most important input.
Worked example: average order value and an employee survey
Switch to Compare two means and load the example: order values have a standard deviation of ₹1,200 and you want to detect a ₹80 increase. You need 3,532 orders in each group. Switch to Survey: a Pune IT services firm with 4,800 employees wants each answer within ±5 points at 95% confidence. Without the correction it would be 385; with it, 356 completed responses, and at a 40% response rate, 890 invitations.
Assumptions
- Independent observations. Randomise by user, not by page view, and count each user once.
- A stable baseline. The baseline rate and standard deviation come from recent, comparable traffic.
- A fixed sample decided in advance. These formulas assume you analyse once, at the planned sample size, not repeatedly as data arrives.
- Random sampling for surveys. The margin of error only covers sampling error, not bias from who chooses to respond.
Getting the same result in Python
from math import sqrt, ceil
from scipy.stats import norm
from statsmodels.stats.power import NormalIndPower, zt_ind_solve_power
from statsmodels.stats.proportion import proportion_effectsize
p1, p2, alpha, power = 0.032, 0.032 * 1.15, 0.05, 0.80
za, zb = norm.ppf(1 - alpha / 2), norm.ppf(power)
pbar = (p1 + p2) / 2
n = (za * sqrt(2 * pbar * (1 - pbar)) + zb * sqrt(p1 * (1 - p1) + p2 * (1 - p2))) ** 2 / (p2 - p1) ** 2
print(ceil(n)) # 22631 per variant (pooled null, unpooled alternative)
h = proportion_effectsize(p2, p1) # Cohen's h, arcsine method
print(ceil(NormalIndPower().solve_power(h, alpha=alpha, power=power, ratio=1))) # 22605
print(ceil(zt_ind_solve_power(effect_size=80 / 1200, alpha=0.05, power=0.80, ratio=1))) # 3532 per group
n0 = norm.ppf(0.975) ** 2 * 0.5 * 0.5 / 0.05 ** 2
print(ceil(n0), ceil(n0 / (1 + (n0 - 1) / 4800))) # 385, 356
Common mistakes
- Peeking and stopping early. Checking every day and stopping at the first p < 0.05 can push the real false positive rate above 20%.
- Choosing the MDE you hope for. An optimistic MDE gives a short test that cannot detect the realistic, smaller effect.
- Mixing up relative and absolute effects. A 1-point lift on a 3% baseline is a 33% relative lift, and needs far fewer users than a 1% relative lift.
- Running for a few days only. Weekday and weekend users behave differently. Run at least one full week, ideally two.