22 A/B Testing Interview Questions & How to Answer Them

We’ve seen candidates ace A/B testing interviews as much as we’ve seen candidates answer interview questions in ways that made them look less qualified than they truly were. So we decided to write this piece to help you do it right, whether you are in product management or data science.

If you’re the hiring manager, you’ll find inspiration for interview questions that’ll help you uncover how good the candidate truly is. If you’re the interviewee, you’ll find what you need to ace the interview.

Key takeaways:

• A/B testing interview questions for product managers should aim to uncover the thought process the candidate applies when making decisions based on the experiment data
• A/B testing interview questions for data scientists should aim to uncover the technical knowledge of statistics and the experiment’s parameters
• Questions that are designed well will go beyond generic A/B testing topics and ask about the company’s particular challenges, such as:
  • the independence assumption when social network effects are present
  • small sample sizes in B2B
  • long-term metrics of A/B testing
• One of the common mistakes in interviews for A/B testing positions is to ask questions that are purely computational questions and do not uncover the candidate’s thought process.

What is A/B testing?

A/B testing, also called split testing, is a method of determining which of two versions of a product feature, web page, campaign element, or other asset performs better for a certain goal. You can use A/B testing to improve the product development workflow, user interface (UI), or conversion rates.

One classic example is to A/B test a button at the bottom of a screen, with two different colors or verbiage, shown randomly between the treatment and the control group. The A/B test would track which button got more clicks. Another example would be to A/B test two flows in a product where the treatment group sees a different feature than the control group. The goal could be to determine which feature leads to a higher in-app spend for two weeks after using the feature.

How to design A/B testing interview questions

Focus on what you’re trying to understand about the candidate’s knowledge and skills

Do you need to know more about how they would design a test, or how they would analyze the results? Do you have a lot of data issues to clean up, or do you need someone comfortable with a specific A/B testing software so they can hit the ground running?

With product management roles, focus on how they make decisions

Probe about how the candidate makes data-driven decisions, combine and use the data while analyzing, or how they apply intuition. For instance, talk about how the candidate would define a minimum detectable effect (MDE), the smallest effect you could have that’s still important for you as a business.

With data science roles, focus on statistical skills

The data scientist on the team will need to be there to ensure that the experiments are set up right. The questions should be about hypothesis testing, confidence intervals, or p-values.

Design questions that are relevant to your company’s day-to-day

Experimentation and analysis are not one-size-fits-all. Each company has different challenges. Focusing on those specific to your own testing context will help you find candidates who understand your industry. For instance, these specific challenges could be that your company:

• Has a small sample size because you’re in B2B
• Is focused on long-term data about the revenue that accumulates over two months after the new feature was first shown
• Has to work around independence assumptions that break because of social network effects

Use case studies

Pair methodology questions with business case questions. The cases should be closely related to your own day-to-day. This further helps in the effort to find out how the candidate would be able to apply general A/B testing knowledge on the specific context of your company.

Ask open-ended questions and make questions incomplete by design

Letting the candidate speak freely while answering the question shows you their thought process. In addition, making the candidate think about what information they’re missing is a way to learn more about how they solve problems in the imperfect world. If you mostly asked questions with one correct answer only, you would not learn enough about the interviewee.

Align your questions with the A/B testing process

Ask questions in the same logical order as the order used in A/B testing. Start with designing the A/B tests before moving into measurement, analyzing results, and making decisions. This approach helps both interviewer and interviewee stay on track when the interview gets longer.

Ask questions that focus on learning ability

Finding a candidate who can adapt and learn fast will help you when you struggle finding someone who can hit the ground running. With the competitive job market, this is more and more important. So ask about:

• How they solved a challenge when they didn’t have all the skills or resources they wished they had
• How they fixed a mistake they made
• How they adapted to a new trend
• What helped them learn a new tool

Discuss industry and A/B testing trends

Assess your candidates’ awareness of how A/B testing evolves for your industry. You’ll get to hear their unique insights while also seeing how they connect their work with the bigger picture. This helps uncover the candidate’s personality, too. You might even have a conversation that you’ll both get passionate about and connect over.

22 real-life examples of A/B testing interview questions — and what to look for in the answers

Below is a representative sample of questions we have asked or have been asked in A/B testing interviews. As a digital optimization company, our questions have a slight bias towards the context of using A/B testing to build and grow a product.

You’ll find questions and answering tips for both product management and data scientist roles, and questions that assume that the hiring process already verified basic knowledge of A/B testing. You’ll also find that the questions apply the tips from the previous section.

Experiment design and setup

You can start your interview by foundational questions about A/B best practices such as:

1. What are ideal conditions for A/B testing?
2. What should you test?
3. If A/B testing is not an available option, how would you answer a question instead?
4. How long would you run an experiment for?

Questions like these will make a capable candidate warm up and feel more comfortable. The answers should then demonstrate that the candidate has both theoretical and practical knowledge, and confirm that you speak the same language.

A/B testing tools

The next batch of questions can focus on the tools of trade:

5. What A/B testing software do you recommend and why, based on your own experience?
6. How would you go about learning a newer A/B testing tool such as Amplitude Experiment?

Great answers will demonstrate that the interviewee has first-hand experience with common tools. If they have a clear answer about how they usually learn a new tool, you’ll know that they’ve done it before. Finally, if they ask a question back and wonder about your own tools, that could indicate they are ready to hit the ground running.

Resolving experimentation issues

Next, consider asking questions about how to deal with problems that come up during experiments:

7. How do you deal with small sample size issues?
8. What issues could impact your A/B test results in the development cycles of our product?
9. How do you mitigate the issues?
10. How do we design the test to minimize interference between control and treatment?

Answers to these questions should come from essential data science knowledge and demonstrate that the candidate understands concepts such as sample size calculators, test timing, or setting up hypotheses that are clear. The answers should also demonstrate that the candidate thinks about mitigating the problems in a structured and proactive manner.

Common A/B testing scenarios

In the final batch of our questions about experiment design and setup, we recommend that you focus on scenarios common in your own organization. For instance:

11. Tell us about a successful A/B test you designed. What were you trying to learn, what did you learn, and how will the experience help you if you work for us?
12. From your experience with using our product, what improvements would you suggest and what experiments would you set up for them?
13. Let’s say we want to compare Feature A and Feature B in an experiment for a user flow. How would you go about designing this test, given what you know about our product?
14. How do you deal with super long term metrics where you have to wait two months to get your metric, for example when you try to test for how much money people spend during the two months after seeing a feature?

Data analysis and decision making

Analysis yields conclusions that vary wildly, depending on the thought process. So ask questions that will help you understand the candidate’s thought process:

15. What would you do if your experiment is inconclusive, and looks more like an AA test? How would you analyze the test results, and what would you look into?
16. When you know there is a social network effect and the independence assumption doesn’t hold, how does it affect your analysis and decisions?
17. In our A/B test, the results were not statistically significant. What are some potential reasons for this?
18. What do you do when you’re testing for two metrics and aim to increase both, but one increases with statistical significance, and the other one decreases with statistical significance?

Great answers will showcase the candidate’s ability to make decisions about rolling out tests, setting up new treatment and control groups, processing conflicting evidence, or trade-off’s between metrics. One viable approach we like to see in interviews is to think about it like finding bugs in engineering, except from a data scientist’s perspective.

Workflows and resources

Our list of recommended questions will conclude with a batch about how the candidate manages the workflow and resources:

19. What software do you recommend for reporting on experiment results?
20. What tools would you integrate with your A/B testing software in order to leverage the experiment data?
21. What new hires would you suggest for your A/B testing team if you already have team members for roles X, Y, Z?
22. Which roles on your product team should be involved in your tests, and how would you make it easy for them to be involved?

The answers to these should prove relevant experience of a product manager, and show that the candidate is organized.

Common A/B testing question mistakes to avoid

If you’ve read this far, you’re familiar with how to design A/B testing interview questions, and you have a list of sample questions that will give you a head-start. If you also avoid the following mistakes we commonly see in interviews, you’ll be on your way to make the hiring or getting hired successful:

Common mistakes by hiring managers when asking the questions:

• Focusing too much on computational questions and not finding out enough about the candidate’s knowledge of the industry, use cases, or processes
• Asking questions with one correct answer instead of asking open-ended questions that would let the candidate express themselves
• Asking generic questions that do not give opportunity to discuss your organization’s context

Common mistakes by candidates when answering the questions:

• Showing your technical skills but not showing your creative side or your analytical thought process
• Talking about your previous experience without making the answers pertinent to the context of the company that’s hiring you
• Focusing on just one tool you used, without showing interest in learning new tools

Pair the new A/B testing position with top tools

A new hire or job will empower you to do great work, if you also have the best software. We invite you to keep going and learn how to analyze A/B test results in Amplitude Analytics, or how to run tests in Amplitude Experiment. You can also review our list of 11 top A/B testing tools out today.

References: