AB Testing Calculator | Calculate Net Promoter Score – Voxco

The first stage in A/B testing (or finding out statistical significance) is to formulate a hypothesis. There is a null hypothesis (H0, 0 being ‘nought’) and an alternative hypothesis. Typically the null hypothesis states that there is no relationship between the variables that you are comparing. The alternative hypothesis tries to prove a relationship exists and that the “test” is successful.

In A/B testing, there can be many instances that would work as a hypothesis – like adding a button on a website or an app, changing the UI or layout or color scheme and testing if these changes are affecting conversion rates by showing some users the normal (control) version. 

A z-score is used to test the veracity of your null-hypothesis. 

A p-value signifies the strength you have in favor of your hypothesis.

In A/B testing you must also decide if you are going to conduct a one-tailed or two-tailed test. One tailed tests can only account for directional effects from your alternative hypothesis. Two tailed tests on the other hand, also account for the eventuality that your hypothesis may have a negative impact. It is the safer approach.