Bayesian A/B Testing Calculator – by Dynamic Yield
Variation A
is the winner!
Based on 95% significance level
Each variation’s long-term probability to out-perform all other live variations, given collected data since the creation or change of any variation included in the test
Probability to be Best
96.79%
3.21%
0%
10%
20%
30%
40%
50%
60%
70%
80%
90%
100%
Samples
Conversion
Conversion Rate
Each variation’s long-term probability to out-perform all other live variations, given collected data since the creation or change of any variation included in the test
Probability to be Best
Assuming I declare the variation as a winner, and I am wrong, how much am I expected to lose in the long term, in term of % vs the variation which is actually the best
Expected Loss
A
8,500
1,500
17.647%
96.79%
0.9%
B
8,500
1,410
16.588%
3.21%
6.22%
The distribution of conversion rates given the sample size collected so far
Posterior simulation of difference
120%
110%
100%
90%
80%
70%
60%
50%
40%
30%
20%
10%
0%
0%
10%
20%
30%
40%
50%
60%
70%
80%
90%
100%
About this tool
You can use this Bayesian A/B testing calculator to run any standard hypothesis Bayesian equation (up to a limit of 10 variations). To do so, specify the number of samples per variation (users, sessions, or impressions depending on your KPI) and the number of conversions (representing the number of clicks or goal completions). Click the Calculate button to compute probabilities.
Please note: This tool does not intend to represent, nor replace Dynamic Yield’s product calculations. Calculations in this tool are based only on binary models, while Dynamic Yield’s product calculations use a different formula for non-binary, revenue-based experiments as well as for handling probabilities for unique conversions.
About DY Labs
DY Lab products are still in development but show a lot of promise, so we want to give you a chance to try them out. These products haven’t been subjected to the same level of reliability, scalability, and security measures as other Dynamic Yield products, and are not being officially supported.
Explore DY Labs →
The Power of Bayesian A/B Testing →
Bayesian A/B Test Duration & Sample Size Calculator →
Frequentism and Bayesianism: A Practical Introduction →
The Importance of Statistical Significance in A/B Tests →
How Not To Run An A/B Test →
Definition of Probability to Be Best in A/B Testing →
