About
The Product Testing questions assess your ability to identify which aspects of a feature to test and which techniques to use. An example of a product testing question is “How would you test LinkedIn’s People You May Know’ feature?”
Evaluation
The interviewer is evaluating you on the following:
- Do you start with the product goal?
- Do you identify what to test and why?
- Do you understand how to use common testing techniques like A/B testing?
- Is your answer structured, or do you tend to talk about random points without a coherent thread?
Product Testing Framework
Structure your answer in the following way:
- Start by describing what the feature does.
- State what the goal of the feature is.
- Describe the product’s different components and explain what they do.
- Discuss design alternatives for those components.
- Hypothesize alternative design versions to compare with current versions. Then state how to test each hypothesis.
- Summarize your analysis and recommend which components to test.
Example
Question
How would you test LinkedIn’s People You May Know feature?
Below is a cursory example answer to this question. You can read the detailed answer from the case “How would you test LinkedIn’s ‘People You May Know’ feature?”
Answer
Product
The ‘People You May Know’ feature displays a list of connection suggestions, where the order in which the individuals appear is prioritized based on certain criteria. The order depends on the recency and how these suggestions are connected to the user. The connection can be a coworker or someone known through other organizations.
Goal
The goal of the feature is to encourage LinkedIn users to connect with the suggestions to grow their professional network.
Alternatives to Test Against
The People You May Know feature has several components we could try to improve and test their impact against the goal.
One idea is to use thumbnails instead of text links to display new suggested connections. Thumbnails will help the user decide whether to connect faster.
Another idea is to show suggested connections from the user’s email or phone contacts when the user is brand new. New users may not have an extensive enough network to suggest secondary connections.
And lastly, the final idea is to show the user more relevant information about the suggested contacts, such as their shared professional interests or skills. This information would be more valuable than how they know the suggested connection.
Test
I would run an A/B test for each idea to see if they impact the goal. The test metric is the number of clicks to make new connections through PYMK. Both the control and treatment groups would include a mix of new users, not so new and old. This will eliminate biases regarding the number of connections users may already have. And I would strive for a significant lift to the test metric, like 5%, before deciding to go ahead with the feature.