Irish Calculators
Business

A/B Conversion Rate Significance Calculator

Compare binary conversion outcomes for two independently assigned visitor groups. The test assumes one outcome per visitor, adequate counts and a testing horizon chosen in advance.

Runs privately in your browserMethod reviewed

Your numbers

Your figures stay on this device.

How the calculation works

An A/B conversion test compares the fraction of visitors converting in two independent groups. This calculator reports the observed effect, a two-sided proportion-test p-value and a 95% interval, rather than turning a difference in raw conversion counts into a claim of improvement.

z = (pB − pA) / sqrt(pooled p × (1 − pooled p) × (1/nA + 1/nB)). The two-sided p-value is 2 × [1 − Φ(|z|)].

  1. Check the experimental design

    Confirm independent random assignment and a testing horizon chosen in advance. Repeated observations, clustered users or selective stopping change the uncertainty this model is designed to represent.

  2. Calculate rates and effect size

    Divide converted visitors by visitors in each group. Subtract A's rate from B's for the percentage-point difference; relative uplift instead divides that difference by A's rate.

  3. Assess uncertainty

    The hypothesis test pools both groups' conversion experience to estimate uncertainty under equal rates. The 95% interval uses an unpooled standard error for the difference, so it serves a different calculation.

Worked example: Equal rates

Sample inputs demonstrate the method. They are not a forecast or product offer.

The first example has 100 conversions from 1,000 visitors in A and 200 from 2,000 in B. Both rates are 10%, so the difference is zero and the two-sided p-value is 1, despite B having twice as many conversions.

Visitors in A
1000
Converted visitors in A
100
Visitors in B
2000
Converted visitors in B
200

Expected result

Conversion rate A
10%
Difference B minus A
0
Two-sided p-value
1
Worked example: Twenty percent uplift

Sample inputs demonstrate the method. They are not a forecast or product offer.

Visitors in A
1000
Converted visitors in A
100
Visitors in B
1000
Converted visitors in B
120

Expected result

Conversion rate B
12%
Relative uplift
20%
Difference B minus A
2

Understanding your result

A small p-value indicates that a difference at least this extreme would be unusual under the equal-rate model and its assumptions. It is not the probability that B is better or the probability that the null hypothesis is true.

Read the interval and effect size alongside the p-value. An interval spanning zero leaves both positive and negative differences compatible with this approximation; it does not establish that the variants are equivalent.

Questions you might have

Can I keep checking until the p-value falls below 0.05?

That changes the false-positive rate. This fixed-horizon test does not implement sequential testing; decide the stopping rule before reading results.

Choosing the right inputs

Visitors in A
Enter eligible visitors assigned to A, counted once each. Use a comparable exposure window for B and exclude repeated sessions counted as separate people.
Converted visitors in A
Count visitors in A who converted, not the number of conversion events. A visitor who purchases twice still contributes one converted visitor to this binary test.
Converted visitors in B
Apply the same conversion definition to B. Each group needs sufficient observed and expected conversions and non-conversions; sparse data are rejected rather than reported with an unsuitable approximation.

Common mistakes to avoid

  • Do not compare raw conversions when group sizes differ.
  • Do not repeatedly check until a desired threshold appears; this is not a sequential-testing procedure.

Assumptions to check

  • Independent random assignment, one binary outcome per visitor, and a pre-specified testing horizon.
  • Normal approximations require sufficiently large conversion and non-conversion counts.

Sources and method

Read our calculation and testing approach.