Skip to main content
Calcimator

A/B Test Calculator

Determine statistical significance of A/B test results. Calculate z-score, p-value, conversion lift, and whether your variant beats the control at your chosen confidence level.

About this calculator

This calculator runs a two-proportion z-test on your control and variant conversion counts and reports whether the difference is statistically significant. Alpha is simply 1 − Confidence Level / 100 (line 34) and responds to nothing else on the page — it's a pure unit conversion of the confidence level you set, not a measured statistic. Significance itself is decided by comparing |Z-Score| against a critical value pulled from a small lookup table that only has exact entries for 90%, 95%, and 99% confidence (line 36); any other confidence level — 92%, for instance, or 97.5% — falls through to the ??

1.96 fallback on line 37 and silently uses the 95% critical value regardless of what you actually chose. Relative Lift compares the variant's conversion rate against the control's and does not depend on your chosen confidence level at all — Confidence Level only moves the significance verdict (Yes/No) and Alpha, never the rates or the lift percentage themselves. The calculator does not correct for multiple comparisons or peeking at results mid-test; it assumes a single, pre-planned comparison between exactly two groups.

Inputs

Results

Statistically Significant?

No

Winner

No clear winner

Control Conv. Rate5%
Variant Conv. Rate6.5%
Relative Lift30%
Z-Score1.44
P-Value0.15
Alpha0.05
How to Use This Calculator
  1. Enter the number of visitors and conversions for both the control and variant groups.
  2. Set your desired confidence level (typically 95%).
  3. Review whether the result is statistically significant and which variant wins.
  4. Check the relative lift percentage to assess practical significance beyond statistical significance.
  5. Only implement the variant if both statistical significance and minimum detectable effect are achieved.

What each input means

Control Visitors
Total number of visitors (or users) in the control group.
Control Conversions
Number of conversions (signups, purchases, clicks) in the control group.
Variant Visitors
Total number of visitors in the variant (test) group.
Variant Conversions
Number of conversions in the variant group.
Confidence Level (%)
The confidence level for determining significance. 95% is standard; 90% is lenient; 99% is strict.

How this is calculated

Worked example, using the default values

  1. Identify Input Parameters
    4 parameters
    Control Visitors = 1000, Control Conversions = 50, Variant Visitors = 1000, Variant Conversions = 65 = 5 input(s) provided
  2. Calculate Statistically Significant?
    No = No
  3. Calculate Winner
    Winner
    No clear winner = No clear winner
  4. Calculate Control Conv. Rate
    Control Conv. Rate
    5 = 5
  5. Calculate Variant Conv. Rate
    Variant Conv. Rate
    6.5 = 6.5

Engine last updated . Checked against 1 independently-derived test — how we verify calculators. Built by Paul Gunder, a software engineer, not a licensed financial, medical, or legal professional.

Frequently Asked Questions

Does Confidence Level change the Relative Lift percentage?

No. Relative Lift is computed purely from the control and variant conversion rates (line 12) and never reads Confidence Level. Confidence Level only changes the critical z-value used to decide Statistically Significant? and the Alpha value shown alongside it.

What critical z-value does the calculator use if I set Confidence Level to 97%?

The lookup table only has exact entries for 90%, 95%, and 99% (line 36); any other value, including 97%, falls through to the "?? 1.96" fallback on line 37 and silently uses the 95% critical value of 1.96 — so 97% and 95% are judged by exactly the same significance bar despite showing different Alpha values.

Is Alpha calculated from my test data?

No. Alpha is just 1 − Confidence Level / 100 (line 34), a direct unit conversion of the confidence level slider — it does not depend on Control Visitors, Control Conversions, Variant Visitors, or Variant Conversions in any way, only on the confidence level you chose.

Do more control conversions make the variant look better or worse?

Worse, all else equal. Relative Lift compares the variant's rate against the control's rate (line 12), so raising Control Conversions raises the control's conversion rate and shrinks the calculated lift, making the variant's improvement look smaller even though its own numbers haven't changed.

Related Calculators

The questions that sit next to this one — chosen by subject, including calculators filed under a different category.

Data Science

Cohort Analysis Calculator

Analyze user cohort retention over time. Calculate monthly retention rates, churn rates, estimated customer lifetime value (LTV), and total cohort revenue.

Data Science

Confusion Matrix Analyzer

Analyze classification model performance from confusion matrix values. Calculate precision, recall, F1 score, MCC, accuracy, specificity, and more.

Statistics

T-Test Calculator

Calculate the t-statistic, degrees of freedom, and approximate p-value for a one-sample t-test.

Streaming & Content

Thumbnail A/B Test Calculator

Compare click-through rates with statistical significance testing.

Construction & Building Trades

Insulation Calculator

Calculate insulation costs, energy savings, and payback period for a project you've already sized -- enter your own expected energy savings percentage.

More in Math & Statistics.