Here’s everything you need to know about A/B testing interview questions in data science interviews.

One of the most common types of data science interview questions from top companies revolves around the humble A/B test. They’re an important test for businesses – A/B tests help businesses decide anything from which product to build to how to design a UI.

The A/B test, also called a split test, is a controlled experiment. You basically split a group of people, like your users or end customers, in half. Offer one half Option A. Offer the other half Option B. Now see which option performs better.

For example, Instagram might want to know if making the Shopping page more prominent would affect the overall spending of users. A team of data scientists would put together a test by selecting a subset of existing Instagram users and testing the original design and the new design to see if the new design offers an improvement in spending.

(Another more recent example is when Instagram changed their feed to be fullscreen – users in the test group hated it!)

A/B tests are critical for companies. This makes it an absolutely vital concept to understand if you’re about to go into a data science interview. You’ll be expected to be confident in the statistical concepts behind an A/B test, how to design a good experiment, and how to interpret the results.

Here’s everything you need to know about A/B testing interview questions in data science interviews.

How to Solve an A/B Test Question

I gave a brief description above, but I want to use this section to investigate the concept deeper. When you get asked about A/B tests in your data science interview, your interviewer will want you to demonstrate both conceptual understanding and your business acumen.

Let’s talk about concepts first. A/B testing is A type of statistical and two-sample hypothesis testing. When you do statistical hypothesis testing, you’re comparing a sample data set against the population data. Two-sample hypothesis testing is when you use statistics to see if the difference between the two samples is significant or not.

When you set up an A/B test, you need two hypotheses: your null hypothesis, and your alternative hypothesis. Your null hypothesis says any change you observe is purely from chance. Basically, it means you think there will be no difference between your control group and your test group.

Your alternative hypothesis says there will be a statistically significant difference between your control and test group.

To develop your hypothesis, you should use the PICOT method: Population, Intervention, Control, Outcome, and Time.

Here’s a template you can use:

In _______(P), what is the effect of _______(I) on ______(O) compared with _______(C) within ________ (T)?

Here’s that Outlook example in the PICOT format:

In Instagram users (P), what is the effect of making the Shopping page more prominent (I) on spending (O) compared with the unchanged Shopping page (C), over the course of three months (T)?

You will have to develop criteria for your control and test group, too. You probably won’t have the resources to do your A/B test on the entire population of users, so you need to come up with a good way to select a sample population. The key is to sample randomly to ensure you have significant results.

If you accidentally select a group of new users who all happen to be millionaires, that will skew your results independently of the test you’re trying to conduct.

You can use Laerd Dissertation’s great guide to selecting a random sample. Make sure your sample is big enough to be statistically relevant. I recommend Andrew Fisher’s guide on the subject.

Once you create your hypotheses, pick your samples, and run your test, it’s time to analyze your results. Before you recommend making a wholesale change, you want to be absolutely sure it’s worth it.

Let’s say you tested your hypothesis on 1,000 users per group. You find that your control group has an average spend of $50, with a standard deviation of $15. Your test group has a retention rate of $53, with a standard deviation of $16.

That may not seem like very much, but with such large sample sizes you can be extremely confident in rejecting your null hypothesis – that gives us a p-value (two-tailed) = 0.000016, which means there’s only a 0.0016% chance that the outcome is through luck or chance.

There are tons of different factors when it comes to choosing the appropriate test. If you have a small sample size (n<30) you’ll choose a t-test rather than a z-test. If you believe your sample mean could be greater, less than, or the same as your control mean, you’ll choose a two-tailed test. I’m a fan of Antoine Soetewey’s flowchart to make a decision. Reviewing this chart will give you a comprehensive overview of the assumptions behind A/B tests and their designs.

Caveats Around A/B Testing

As you answer these types of questions, you’ll likely be tested on your flexibility of thought. A/B tests are great, but the circumstances often aren’t ideal. Employers will want to know how you handle challenges.

If you’re in B2B, you’ll likely face smaller sample sizes. You may not be able to assume independent populations due to the social network effects. You may not have the resources or time to design a perfect test. There might be times when it doesn’t make sense to run an A/B test.

Interviewers will want to hear how you plan to face those challenges as much as your mastery over statistical tests.

You will face questions such as:

• How will you plan to compensate for a small sample size?
• How will you minimize interference between control and test groups?
• How will you make a test robust if you can’t run it for as long as you want?

Examples of A/B Test Questions

Let’s solve a real-world A/B test question example from StrataScratch platform:

Robinhood wants to encourage more people to create an account and plans a campaign where they would give out a few free stocks to new users. Describe in detail how you would implement an A/B test to determine the success of this campaign.

When you answer this type of interview question, you’ll want to address the issues in order:

• Make any clarifying statements or questions.
• What does “a few stocks” mean in this instance? Would it be possible to test multiple numbers of free stocks?
• What metric measures success? Retention, money spent, anything else?
• Which stock to choose? Or offer any stock of X value?
• Determine:
  • The length of your experiment. 
  • How you’ll select your sample.
  • The size of your sample.
  • For these questions, there’s no right answer. Just your thought process on how you would determine these.
• Develop your hypothesis:
  • In new Robinhood users (P), what is the effect of free stocks (I) on your success metric (O) compared with not giving new signups free stocks (C) within your selected time frame (T)?
• Run the analysis.
• Measure the results:
  • What test would you use and why? 
  • What qualifies as success?