How Are You Being Evaluated
This product feature testing question checks to see if you can identify which aspects of a feature to test and which techniques to use.
The interviewer is evaluating you on the following:
- Do you start with the goal?
- Do you identify what to test and why?
- Do you understand how to use common testing techniques like A/B testing?
- Is your answer structured or do you tend to talk about random points without a coherent thread?
Answer Structure
Structure your answer in the following way:
- Start by describing what the feature does.
- State what the goal of the feature is.
- Talk about the different components and describe what they do.
- Discuss design alternatives for those components.
- Hypothesize alternative design versions to compare with current versions. Then state how to test each hypothesis.
- Summarize your analysis and recommend which components to test.
Answer Example
INTERVIEWEE: I’ve seen this feature on the My Network page of LinkedIn. Before beginning, I would like to confirm my understanding of how this feature works.
INTERVIEWER: Sure.
INTERVIEWEE: The ‘People You May Know’ feature displays a list of connection suggestions, where the order in which the individuals appear is prioritized based on certain criteria. The order depends on how these suggestions are connected to the user. The connection could be via a mutual co-worker or co-workers, employers, schools attended, membership in organizations, or it could be an email or phone contact. And, it also seems that individuals associated with recent events that happened to the user are listed first. For example, if a user just recently added a new person to his network, than the list will display connection suggestions that are in the new person’s network.
INTERVIEWER: That is a simplification but is close enough. Time overlap with a person in a company or other institution is also a key component in the prioritization. But, let’s go with your description. So now, how would you test the ‘People You May Know’ feature?
INTERVIEWEE: Okay. I imagine the overall goal of this feature is to encourage LinkedIn users to grow their network by connecting to more people. Is this correct?
INTERVIEWER: Yes.
INTERVIEWEE: Ok. If I were the product manager, I would test alternative ideas against three components: the format for presenting connection suggestions, the order suggestions are listed, and the information shared about a possible connection.
Testing format of presenting connection suggestion information
Let’s start with the format of presenting a connection suggestion. Currently, the feature displays a thumbnail image along with three lines of content. The first line is a name, the second line is a title, and the third line is a link that shows how this possible connection is related to the user. This link is a chain link icon, followed by the name of a user’s top mutual connection and a number indicating more mutual connections. For example, the text could show John Doe and 7 others. Clicking on this link takes the user to a list that displays details about those mutual connections.
One question I have is if instead of using a text link to mutual connections, what if we used thumbnail images of the mutual connections? The photos would be a more visual and direct way of highlighting them. A text link such as “…and 7 others” requires an extra step to discover who those seven connections are. And, it is generally better to decrease friction by reducing the number of steps.
I would test this hypothesis using A/B testing. Version A would be the current solution with the text link, and version B would be the alternative solution using thumbnail images of the mutual connections. In case there are too many mutual connections to display, a link can be added to view additional images. If version B results in a higher number of clicks on the Connect button, then the thumbnail images version would be the better choice.
Testing Prioritization
I believe the algorithm for choosing and picking the order in which connection suggestions appear works well when the user already has many existing connections. But the algorithm may not perform as well if the user is new and does not have many connections. One question I have is whether, in the case of a newly registered user, it is better to use the user’s email and phone contacts not only as connection suggestions but also as leads to additional suggested connections. Using the newly registered user’s contacts may lead to more recommendations.
Again, I would use A/B testing to test this hypothesis. Version A representing the current way connection suggestions are found, and version B representing the way they are found based on email and phone contacts. This should be tested on newly registered users, not existing ones. If the Connect button in version B is clicked on with higher frequency, this would mean that showing a suggestion connection using email and phone contacts is a better approach than the default scheme for the case of newly registered users.
Currently, there is not an option to filter connection suggestions based on a mutual connection, or company, or school. One idea is to let people filter connection suggestions based on these three categories. To test this, I would run an A/B test where version A would be the current list of connection suggestions with no filter and version B would be a list of connection suggestions with the filter option. If the result of the test shows that version B received more clicks on the Connect button, then version B would be the better option.
Testing the amount of new contact information
Finally, let’s talk about the information displayed about the connection suggestions. Currently, only three lines of information are shown, the name of the person, the title and a link that shows how the connection suggestion was found. The suggestion was made either because of mutual connections, mutual workplace or institution. Perhaps providing an additional data point like shared skills could tip the user’s decision to click on the Connect button. So I would do an A/B test, where version A represents the current way of displaying information about a connection suggestion, and version B would have the same information as version A but with an additional line that shows shared professional skills. If the Connect button is clicked more frequently on version B, then adding shared skills would be beneficial.
To summarize, I would test the ‘People You May Know’ feature by first decomposing it into core elements: the format in which the connection suggestions are displayed, the prioritization criteria behind the order of connection suggestions, and the amount of information displayed about a connection suggestion. Of all the suggestions I made for testing this feature, I would prioritize testing the filter idea and including shared ‘skills’ to the information presented about the connection suggestion. The ability to filter data in a list has become so common that not having it may frustrate users. And, people looking for jobs are likely to want to network with others that have similar skills. Therefore, testing the appeal of these alternative suggestions would be my first priorities.