Statistical Significance Calculator | Z-Score & Confidence Levels | Evolytics

Calculate the Statistical Significance of Your A/B Test

Z-Score Calculator

Recipe Sample Size

Control Sample Size

Recipe Response %

Control Response %

Z-score:

Statistically Significant:

Raw Difference:

Index to Control:

95% Confidence Interval

Min
Max

Raw Percent Difference

Variant Recipe Response Rate

95% confidence interval means that we are 95% confident the variant mean falls
between these two values.

Frequently Asked Questions

What is a Z-Score?

A z-score is a standardized score that describes how many standard deviations an element is from the mean.

In A/B Testing terms, all of your visitors are observations, and the Control experience makes up a bell curve. The Variant Recipe and all of the visitors in it make up a second bell curve. We use the Z-score calculator to test how far the center of the Variant bell curve is from the center of the Control bell curve.

Is My Test One-Sided or Two-Sided?

We typically recommend two-sided tests. If you conduct a two-sided hypothesis test, you can be mathematically confident about whether or not your Variant Recipe is greater than or less than your Control Recipe. With a one-sided test, you are only mathematically confident about one or the other, but never both. We believe it’s just as important to know if your test is statistically underperforming as it is to know if it’s performing better than Control.

What Does My Confidence Level Mean to Me in a Business Sense?

Z-scores are equated to confidence levels.

If your two-sided test has a z-score of 1.96, you are 95% confident that that Variant Recipe is different than the Control Recipe. If you roll out this Variant Recipe, there is only a one in 20 chance that you will not see a lift.

What Are Common Confidence Levels?

The most commonly used confidence level is 95%. This is the standard confidence level in the scientific community, essentially stating that there is a one in twenty chance of an alpha error, or the chance that the observations in the experiment look different, but are not.

Common Confidence Levels and Their Z-Score Equivalents

• 95%
  • Two-Sided Z-Score: 1.96
  • One-Sided Z-Score: 1.65
• 99%
  • Two-Sided Z-Score: 2.58
  • One-Sided Z-Score: 2.33
• 90%
  • Two-Sided Z-Score: 1.64
  • One-Sided Z-Score: 1.28

In the digital community, it’s not uncommon to see A/B testing tools make calls at only 80% or 85% confidence. While there are a limited set of situations when this is okay, it is never ideal. Making decisions too early is one of the most common mistakes we see in A/B Testing.

If you make ROI projections based on 80% confidence and roll out that experience, you have a one in five chance of missing them completely. If you do one test a month, at least two likely had erroneous results.

Of course, we don’t recommend waiting for 99% confidence either. We recommend setting standards based on available traffic levels, risk appetite, and the willingness to back test.

DISCOVER OTHER HELPFUL RESOURCES

See All Resources

Chi Square Calculator

Determine if the applied recipe influences conversion by measuring the distance of the actual counts from those that would be expected if they were not related.

Learn More

Sample Size Calculator

Calculate the sample size necessary to reach 95% confidence with your current conversion rate and desired lift.

Learn More

Z-Score to Confidence Calculator

Using your standardized z-score, you can use this calculator to convert that score to a confidence level for either one-sided or two-sided A/B tests.

Learn More

Related A/B Testing Blog Posts

A/B testing and experimentation insights and best practices.

See All

Measuring Success in A/B/n Testing Programs

The Whole is Greater than the Sum of its Parts The results of your latest A/B/n test are in-hand, they’re…

Learn More

What’s the Optimal Success Rate for an A/B Testing Program?

Why You Should Aim For a 20-40% Win Rate As more businesses harness the power of experimentation to enhance user…

Learn More

Migrate to One of These A/B Testing Platforms Before Google Optimize Sunsets

Google will discontinue its Optimize A/B testing platform on September 30, 2023. We’ll help you migrate and maximize your testing…

Learn More

Go Ahead, Take a Peek at Your Testing Data!

Implementing Sequential Testing in an A/B/n Testing Program To Peek or Not to Peek In experimentation, peeking at the results…

Learn More

Sign up to receive analytics tips and insights straight to your inbox.

By entering your email address, you consent to receive communications from Evolytics. You may unsubscribe at any time by clicking the unsubscribe link located at the bottom of any email. This site is protected by reCAPTCHA and the Google Privacy Policy and Terms of Service apply.