A/B Testing

Sample Size Calculator

Work out how many users you need before you start. Get the sample per variant and the number of days to run an A/B test on conversion rates, the group sizes to compare two means, or the number of responses a survey needs for a given margin of error, with the formula worked through on your numbers.

Your study

Smallest lift worth detecting.

Use the standard deviation of the metric from past data. Δ is in the same units.

Traffic is optional. Add it to get the number of days to run.

50% gives the largest, safest sample. Leave population blank for a large or unknown population.

Fill in your study and press Calculate, or load the example.

What this calculator does

A sample size calculation answers one question before you collect any data: how many observations do you need so that a real effect of a given size is likely to show up as significant? Too few and the test is underpowered, so a genuine improvement looks like noise. Too many and you waste traffic and time that could have gone into the next experiment.

The calculator covers the three cases analysts meet most often. A/B test on conversion rates gives the users per variant, the total, and how many days to run at your traffic. Compare two means handles continuous metrics such as order value or delivery time, with unequal group sizes if you want them. Survey or proportion estimate gives the number of completed responses for a chosen margin of error, with the finite population correction for small populations.

The formulas

Two proportions: n = [ zα √(2 p̄(1 − p̄)) + zβ √(p₁(1 − p₁) + p₂(1 − p₂)) ]² / (p₂ − p₁)²,   p̄ = (p₁ + p₂) / 2
Two means: n₁ = (zα + zβ)² σ² (1 + 1/r) / Δ²,   n₂ = r × n₁
Survey: n₀ = z² p(1 − p) / E²,   with population N: n = n₀ / (1 + (n₀ − 1) / N)
zα = z1 − α/2 (two-sided) or z1 − α (one-sided),   zβ = zpower,   Bonferroni: α / (variants − 1)

Each result is rounded up to a whole person. The proportion formula uses the pooled rate under the null hypothesis and the two separate rates under the alternative, which is the standard normal approximation behind most A/B testing tools.

Worked example: UPI one-tap checkout

Press Load example. A Bengaluru D2C brand converts 3.2% of checkout visitors and wants to test a one-tap UPI payment button. A lift smaller than 15% (3.2% to 3.68%) would not justify the engineering work, so that is the minimum detectable effect. The site gets 8,000 checkout visitors a day and half of them go into the test.

  • Sample: 22,631 visitors per variant, 45,262 in total, at α = 0.05 (two-sided) and 80% power.
  • Duration: 45,262 / 4,000 a day = 11.3, so at least 12 days. Rounding up to two full weeks (14 days) covers each weekday twice.
  • Trade-off: the Scenarios tab shows that a 10% MDE would need about 2.2 times as many visitors, which is why the MDE is the most important input.

Worked example: average order value and an employee survey

Switch to Compare two means and load the example: order values have a standard deviation of ₹1,200 and you want to detect a ₹80 increase. You need 3,532 orders in each group. Switch to Survey: a Pune IT services firm with 4,800 employees wants each answer within ±5 points at 95% confidence. Without the correction it would be 385; with it, 356 completed responses, and at a 40% response rate, 890 invitations.

Assumptions

  • Independent observations. Randomise by user, not by page view, and count each user once.
  • A stable baseline. The baseline rate and standard deviation come from recent, comparable traffic.
  • A fixed sample decided in advance. These formulas assume you analyse once, at the planned sample size, not repeatedly as data arrives.
  • Random sampling for surveys. The margin of error only covers sampling error, not bias from who chooses to respond.

Getting the same result in Python

from math import sqrt, ceil
from scipy.stats import norm
from statsmodels.stats.power import NormalIndPower, zt_ind_solve_power
from statsmodels.stats.proportion import proportion_effectsize

p1, p2, alpha, power = 0.032, 0.032 * 1.15, 0.05, 0.80
za, zb = norm.ppf(1 - alpha / 2), norm.ppf(power)
pbar = (p1 + p2) / 2
n = (za * sqrt(2 * pbar * (1 - pbar)) + zb * sqrt(p1 * (1 - p1) + p2 * (1 - p2))) ** 2 / (p2 - p1) ** 2
print(ceil(n))   # 22631 per variant (pooled null, unpooled alternative)

h = proportion_effectsize(p2, p1)   # Cohen's h, arcsine method
print(ceil(NormalIndPower().solve_power(h, alpha=alpha, power=power, ratio=1)))   # 22605

print(ceil(zt_ind_solve_power(effect_size=80 / 1200, alpha=0.05, power=0.80, ratio=1)))   # 3532 per group

n0 = norm.ppf(0.975) ** 2 * 0.5 * 0.5 / 0.05 ** 2
print(ceil(n0), ceil(n0 / (1 + (n0 - 1) / 4800)))   # 385, 356

Common mistakes

  • Peeking and stopping early. Checking every day and stopping at the first p < 0.05 can push the real false positive rate above 20%.
  • Choosing the MDE you hope for. An optimistic MDE gives a short test that cannot detect the realistic, smaller effect.
  • Mixing up relative and absolute effects. A 1-point lift on a 3% baseline is a 33% relative lift, and needs far fewer users than a 1% relative lift.
  • Running for a few days only. Weekday and weekend users behave differently. Run at least one full week, ideally two.

Related learning on Codeayan

FAQ

Sample size questions

How long should I run my A/B test?

Divide the total sample (n per variant × number of variants) by the visitors who enter the test each day. That is the minimum. Then round up to whole weeks, so every weekday and weekend is covered equally, and do not stop early because the result looks significant halfway through: peeking inflates the false positive rate well above your alpha.

What minimum detectable effect (MDE) should I choose?

Choose the smallest lift that would change your decision or pay for the work, not the lift you hope to see. A relative MDE of 10% on a 3% conversion rate means detecting 3.0% against 3.3%. Halving the MDE roughly quadruples the sample, so if the duration comes out too long, either accept a larger MDE or test a bolder change.

Why does another calculator give a slightly different number?

There are several valid formulas. This calculator uses the normal approximation with the pooled rate under the null and separate rates under the alternative. Tools based on Cohen's h (the arcsine transformation, used by statsmodels and G*Power) and tools that add a continuity correction (Fleiss) typically differ by a few percent, and some round z-values or use one-sided tests by default. The Other methods tab shows all of them side by side for your inputs.

Should I use a one-sided or two-sided test?

Two-sided is the safe default: it detects the variant being better or worse, which you usually want to know. A one-sided test needs about 20% fewer users at 80% power, but only if you decided before the test that you will treat "worse" and "no different" the same way. Switching to one-sided after seeing the data is not allowed.

What does the Bonferroni correction do for several variants?

With one control and three variants you make three comparisons, so the chance of at least one false positive rises well above 5%. Bonferroni divides alpha by the number of comparisons (0.05 / 3 = 0.0167 here), which keeps the overall error rate at 5% but needs more users per variant. It is slightly conservative, which is the safe direction.

Why is 385 the magic number for surveys?

At 95% confidence, a margin of error of ±5 percentage points and the worst-case proportion of 50%, n = 1.96² × 0.5 × 0.5 / 0.05² = 384.1, which rounds up to 385. It assumes a simple random sample from a large population. For a small population, the finite population correction lowers it; for low response rates you need to invite more people to get 385 completed answers.

More tools

Keep going