A/B Testing Calculator – significance calculator for A/B tests

Much faster A/B tests

Run tests 20-80% faster than traditional best-practice methods, without sacrificing the rigorous statistical error control. This results in benefiting from winners earlier and cutting short losers as quickly as possible. The overall effect is an increase in the business returns of your A/B testing program. Real-life data show an average benefit of 26% for clients of Analytics Toolkit.

The above is all possible thanks to the AGILE statistical method pioneered in the industry by the author of this calculator, Georgi Georgiev.

Evaluate data as it gathers

Achieve flexibility and efficiency with your A/B tests by periodically performing significance tests on the data while avoiding unaccounted for peeking. This is accomplished by special stopping boundaries for efficacy and futility which guide you when to call a test a winner or to cut a non-promising test short.

With this A/B test significance calculator there is no longer the need to compromise between running a proper test and pleasing a client or HiPPo who demands to act on the results right this minute.

Calculations you can trust

The unrivaled statistical rigor of this A/B test calculator means you can trust your results are statistically validated. P-values, confidence intervals, and lift estimates are adjusted so they account for testing multiple variants, for periodical evaluation of the data, and other applicable complications.

The tool is embedded in our A/B testing hub making planning and analyzing tests a seamless experience which helps you avoid many of the common pitfalls encountered in applying statistics in A/B testing. Diagnostic tests such as SRM checks are performed automatically to alert you of unforeseen issues. All of this results in trustworthy test results.

Detailed help information

Accessible but exact language with generously sprinkled inline help references and detailed explanations makes navigating the statistical jargon as easy as it can be. Graphical displays of all statistical results help avoid widespread misunderstandings and ensure the correct conclusions are drawn from the statistics.

Experienced CRO professionals and web analysts will appreciate the fairly detailed documentation and explanations accompanying each input and output, giving them the necessary understanding of the underlying machinery and assumptions involved.

Easily input your test data

There are three ways to get data into this statistical calculator. First, you can manually enter summary data. A tedious process for sure, even with file uploads, but it comes with the benefit that it can be used with any testing system.

The second data input method supported by our statistical significance calculator is to use our simple Data API. It can work with any system for storing A/B test data, including custom-built ones, as long as you have the capability to code the API endpoint. It is nicely complitmented by our powerful Reporting API.

Third, if your AB testing delivery software syncs data to MixPanel® our calculator can automatically extract it using the MixPanel® API. This happens in a completely automated fashion, and you will get email notifications when a test is ready to be stopped.