What Is A/B Testing and How Is It Used? | HBS Online

You may have experienced meetings where a lot of ideas are circulated about how to improve an existing product or service. In these meetings, differing opinions can quickly turn into a battle of long-winded defenses. Fortunately, the emergence of A/B testing—once thought to be exclusive to tech firms—has become a viable and cost-effective way for all types of businesses to identify and test value-creating ideas.

Related: The Advantages of Data-Driven Decision Making

What Is an A/B test?

In statistical terms, A/B testing is a method of two-sample hypothesis testing. This means comparing the outcomes of two different choices (A and B) by running a controlled mini-experiment. This method is also sometimes referred to as split testing.

What Is A/B Testing Used For?

A/B testing is often discussed in the context of user experience (UX), conversion rate optimization (CRO), and other marketing and technology-focused applications; however, it can be valuable in other situations as well.

Although the concept of A/B testing was galvanized by Silicon Valley giants, the rationale behind A/B testing isn’t new. The practice borrows from traditional randomized control trials to create smaller, more scalable experiments.

For this reason, professionals also perform A/B testing to gather valuable insights and guide important business decisions, such as determining which product features are most important to consumers.

A/B testing is a popular method of experimentation in the fields of digital marketing and web design. For example, a marketer looking to increase e-commerce sales may run an experiment to determine whether the location of the “buy now” button on the product page impacts a particular product’s number of sales.

In this scenario, version A of the product page might feature the button in the top-right corner of the page while version B places the button in the bottom-right corner. With all other variables held constant, randomly selected users interact with the page. Afterward, the marketer can analyze the results to determine which button location resulted in the greatest percentage of sales.

An Example of A/B Testing

As a basic example, let’s say you’re an abstract artist. You’re confident in your technique but aren’t sure how the outside world—and, more importantly, art critics—will respond to your new paintings. Assessing the quality of art is a famously challenging process.

To employ A/B testing for this scenario, start by creating two different paintings that are alike. As you paint both pieces, change one small thing—for instance, add a red square to one painting and not the other. Again, this means that everything about the paintings is alike except for this one modification. Once the change is made, display the two paintings in randomly selected art galleries across the country and wait for your art agent, or another unbiased third party, to gather reactions and report back.

After each painting has been placed in a reasonable amount of art galleries, the feedback may reflect that the painting with the red square received significantly more praise, or maybe it didn’t. The hypothetical outcome doesn’t matter. Rather, what matters is that you can be reasonably confident that your change will or will not make the painting better, and you can go on to create better art as a result.

“America’s Most Wanted” by Komar and Melamid used a different technique (surveys) to create a painting that catered to the American public’s art preferences. Source: Dia Art Foundation.

The randomization aspect of this design is explicitly emphasized because randomization is the gold-standard for eliminating biases. Art is a subjective field and evolves over time. So do the preferences and opinions of customers, clients, or coworkers. A/B testing isn’t a static process, and tests can be repeated or complemented if companies believe that findings may not be valid or applicable anymore.

As a final note, it’s imperative that A/B testing design be rigorous to ensure the validity of results. Furthermore, there may be some decisions where internal opinions are more cost-effective or timely.

Making Data-Driven Business Decisions

Companies like Google, Amazon, and Facebook have all used A/B testing to help create more intuitive web layouts or ad campaigns, and firms in all industries can find value in experimentation. Using data-driven decision-making techniques empowers business leaders to confidently pursue opportunities and address challenges facing their firms. Customers benefit and companies can reap measurable monetary returns by catering to market preferences.

Do you want to learn how to apply fundamental quantitative methods to real business problems? Explore Business Analytics and our other analytics courses to find out how you can use data to inform business decisions.

This post was updated on January 12, 2021. It was originally published on December 15, 2016.